bug#24315: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

bug#24315: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Peder O. Klingenberg]

[-- Attachment #1: Type: text/plain, Size: 21543 bytes --]


I recently started using gnus-icalendar to handle meeting invitations.
Someone sent me an invitation that included a rather long email thread
in typical outlook style, and gnus errored out when trying to display
the article.

The backtrace showed that the error comes from the call
  (re-search-forward  "\\(.*\\)\\(\r?\n[ \t].*\\)*" nil t)
in icalendar--read-element (nil nil).

I reduced and anonymized the contents of the message, the problematic
section is attached to this bug report.  The problem is reproducible for
me from emacs -Q by the following recipe:

  C-x C-f calendar-event-problem.txt
  <make sure cursor is somewhere in the first couple of lines>
  M-: (re-search-forward  "\\(.*\\)\\(\r?\n[ \t].*\\)*" nil t)

Result:
Debugger entered--Lisp error: (error "Variable binding depth exceeds max-specpdl-size")
  re-search-forward("\\(.*\\)\\(?\n[ 	].*\\)*" nil t)
  eval((re-search-forward "\\(.*\\)\\(?\n[ 	].*\\)*" nil t) nil)
  eval-expression((re-search-forward "\\(.*\\)\\(?\n[ 	].*\\)*" nil t) nil)
  funcall-interactively(eval-expression (re-search-forward "\\(.*\\)\\(?\n[ 	].*\\)*" nil t) nil)
  call-interactively(eval-expression nil nil)
  command-execute(eval-expression)

Reducing the text size by about 1/3, or just moving the cursor a ways
down the buffer before evaling, eliminates the error.

From the same emacs -Q session:

max-specpdl-size is a variable defined in ‘C source code’.
Its value is 2930
Original value was 1300

Increasing max-specpdl-size to 20000 does not help, the problem still
occurs.

Strangely, the same recipe does not reproduce the error on os x, with a
build from master from 2016-08-12, but on my GNU/Linux machine, which is
where I run gnus and need this to work, it is 100% reproducible.  I am
on Kubuntu 14.04.5, emacs is compiled with gcc (Ubuntu
4.8.4-2ubuntu1~14.04.3) 4.8.4.

Is this reproducible for anyone else, or do I need to dig deeper into my
environment?  Any ideas on where to start digging?



In GNU Emacs 25.1.50.2 (x86_64-unknown-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2016-08-11 built on luna
Repository revision: 36a57c55f2b855bca704c26bf7d787b7b471fe16
Windowing system distributor 'The X.Org Foundation', version 11.0.11604000
System Description:	Ubuntu 14.04.5 LTS

Recent messages:
scroll-down-command: Beginning of buffer [3 times]
(Shell command succeeded with no output) [91 times]
Back to top level
When done with this frame, type C-x 5 0
Undo! [3 times]
calendar-event-problem.txt changed on disk; really edit the buffer? (y, n, r or C-h) r
ask-user-about-supersession-threat: File reverted: /home/pok/tmp/calendar-event-problem.txt
previous-line: Beginning of buffer [3 times]
Entering debugger...
Back to top level

Configured using:
 'configure --prefix=/usr/local/emacs-git
 --enable-locallisppath=/etc/emacs24:/etc/emacs:/usr/local/share/emacs/site-lisp/:/usr/share/emacs/site-lisp
 --with-x=yes --with-x-toolkit=lucid --with-toolkit-scroll-bars
 --with-pop=yes'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GCONF
GSETTINGS NOTIFY LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT
ZLIB TOOLKIT_SCROLL_BARS LUCID X11

Important settings:
  value of $LC_MONETARY: en_US.UTF-8
  value of $LC_NUMERIC: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Text

Minor modes in effect:
  csv-field-index-mode: t
  magit-auto-revert-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  shell-dirtrack-mode: t
  diff-auto-refine-mode: t
  show-paren-mode: t
  yas-global-mode: t
  yas-minor-mode: t
  ido-everywhere: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  auto-fill-function: do-auto-fill
  transient-mark-mode: t

Load-path shadows:
/home/pok/.emacs.d/elpa/org-20160822/org-macs hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-macs
/home/pok/.emacs.d/elpa/org-20160822/ob-dot hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-dot
/home/pok/.emacs.d/elpa/org-20160822/ob-mscgen hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-mscgen
/home/pok/.emacs.d/elpa/org-20160822/ob-clojure hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-clojure
/home/pok/.emacs.d/elpa/org-20160822/org-install hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-install
/home/pok/.emacs.d/elpa/org-20160822/org-faces hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-faces
/home/pok/.emacs.d/elpa/org-20160822/ob-scheme hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-scheme
/home/pok/.emacs.d/elpa/org-20160822/ob-haskell hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-haskell
/home/pok/.emacs.d/elpa/org-20160822/ob-sqlite hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-sqlite
/home/pok/.emacs.d/elpa/org-20160822/ox-texinfo hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-texinfo
/home/pok/.emacs.d/elpa/org-20160822/ob-perl hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-perl
/home/pok/.emacs.d/elpa/org-20160822/ob-makefile hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-makefile
/home/pok/.emacs.d/elpa/org-20160822/ob-io hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-io
/home/pok/.emacs.d/elpa/org-20160822/org-docview hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-docview
/home/pok/.emacs.d/elpa/org-20160822/org-irc hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-irc
/home/pok/.emacs.d/elpa/org-20160822/ob-C hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-C
/home/pok/.emacs.d/elpa/org-20160822/ox-man hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-man
/home/pok/.emacs.d/elpa/org-20160822/org-capture hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-capture
/home/pok/.emacs.d/elpa/org-20160822/ob-table hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-table
/home/pok/.emacs.d/elpa/org-20160822/ob-sql hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-sql
/home/pok/.emacs.d/elpa/org-20160822/org-list hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-list
/home/pok/.emacs.d/elpa/org-20160822/org-crypt hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-crypt
/home/pok/.emacs.d/elpa/org-20160822/org-attach hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-attach
/home/pok/.emacs.d/elpa/org-20160822/org-colview hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-colview
/home/pok/.emacs.d/elpa/org-20160822/org-id hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-id
/home/pok/.emacs.d/elpa/org-20160822/ob-plantuml hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-plantuml
/home/pok/.emacs.d/elpa/org-20160822/ox-publish hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-publish
/home/pok/.emacs.d/elpa/org-20160822/ob-tangle hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-tangle
/home/pok/.emacs.d/elpa/org-20160822/org hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org
/home/pok/.emacs.d/elpa/org-20160822/org-indent hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-indent
/home/pok/.emacs.d/elpa/org-20160822/ox-ascii hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-ascii
/home/pok/.emacs.d/elpa/org-20160822/ob-matlab hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-matlab
/home/pok/.emacs.d/elpa/org-20160822/org-gnus hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-gnus
/home/pok/.emacs.d/elpa/org-20160822/org-feed hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-feed
/home/pok/.emacs.d/elpa/org-20160822/org-clock hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-clock
/home/pok/.emacs.d/elpa/org-20160822/org-element hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-element
/home/pok/.emacs.d/elpa/org-20160822/ob-screen hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-screen
/home/pok/.emacs.d/elpa/org-20160822/ob hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob
/home/pok/.emacs.d/elpa/org-20160822/org-mobile hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-mobile
/home/pok/.emacs.d/elpa/org-20160822/ox-latex hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-latex
/home/pok/.emacs.d/elpa/org-20160822/ob-lisp hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-lisp
/home/pok/.emacs.d/elpa/org-20160822/ob-calc hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-calc
/home/pok/.emacs.d/elpa/org-20160822/org-bibtex hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-bibtex
/home/pok/.emacs.d/elpa/org-20160822/org-macro hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-macro
/home/pok/.emacs.d/elpa/org-20160822/org-agenda hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-agenda
/home/pok/.emacs.d/elpa/org-20160822/org-entities hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-entities
/home/pok/.emacs.d/elpa/org-20160822/ob-css hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-css
/home/pok/.emacs.d/elpa/org-20160822/ob-R hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-R
/home/pok/.emacs.d/elpa/org-20160822/org-mouse hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-mouse
/home/pok/.emacs.d/elpa/org-20160822/org-timer hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-timer
/home/pok/.emacs.d/elpa/org-20160822/ob-java hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-java
/home/pok/.emacs.d/elpa/org-20160822/ob-gnuplot hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-gnuplot
/home/pok/.emacs.d/elpa/org-20160822/ob-keys hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-keys
/home/pok/.emacs.d/elpa/org-20160822/ob-python hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-python
/home/pok/.emacs.d/elpa/org-20160822/org-compat hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-compat
/home/pok/.emacs.d/elpa/org-20160822/ob-fortran hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-fortran
/home/pok/.emacs.d/elpa/org-20160822/ob-exp hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-exp
/home/pok/.emacs.d/elpa/org-20160822/ox-html hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-html
/home/pok/.emacs.d/elpa/org-20160822/ob-comint hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-comint
/home/pok/.emacs.d/elpa/org-20160822/org-src hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-src
/home/pok/.emacs.d/elpa/org-20160822/ob-scala hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-scala
/home/pok/.emacs.d/elpa/org-20160822/org-version hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-version
/home/pok/.emacs.d/elpa/org-20160822/org-loaddefs hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-loaddefs
/home/pok/.emacs.d/elpa/org-20160822/ob-emacs-lisp hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-emacs-lisp
/home/pok/.emacs.d/elpa/org-20160822/org-datetree hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-datetree
/home/pok/.emacs.d/elpa/org-20160822/ob-asymptote hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-asymptote
/home/pok/.emacs.d/elpa/org-20160822/ox-org hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-org
/home/pok/.emacs.d/elpa/org-20160822/org-habit hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-habit
/home/pok/.emacs.d/elpa/org-20160822/ox hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox
/home/pok/.emacs.d/elpa/org-20160822/org-w3m hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-w3m
/home/pok/.emacs.d/elpa/org-20160822/ob-lob hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-lob
/home/pok/.emacs.d/elpa/org-20160822/org-protocol hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-protocol
/home/pok/.emacs.d/elpa/org-20160822/ob-shen hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-shen
/home/pok/.emacs.d/elpa/org-20160822/ob-maxima hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-maxima
/home/pok/.emacs.d/elpa/org-20160822/ob-ref hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-ref
/home/pok/.emacs.d/elpa/org-20160822/ob-org hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-org
/home/pok/.emacs.d/elpa/org-20160822/ob-latex hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-latex
/home/pok/.emacs.d/elpa/org-20160822/ox-md hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-md
/home/pok/.emacs.d/elpa/org-20160822/org-mhe hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-mhe
/home/pok/.emacs.d/elpa/org-20160822/ob-ledger hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-ledger
/home/pok/.emacs.d/elpa/org-20160822/org-archive hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-archive
/home/pok/.emacs.d/elpa/org-20160822/org-ctags hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-ctags
/home/pok/.emacs.d/elpa/org-20160822/org-info hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-info
/home/pok/.emacs.d/elpa/org-20160822/ob-octave hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-octave
/home/pok/.emacs.d/elpa/org-20160822/org-pcomplete hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-pcomplete
/home/pok/.emacs.d/elpa/org-20160822/ob-ditaa hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-ditaa
/home/pok/.emacs.d/elpa/org-20160822/org-plot hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-plot
/home/pok/.emacs.d/elpa/org-20160822/ob-eval hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-eval
/home/pok/.emacs.d/elpa/org-20160822/org-table hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-table
/home/pok/.emacs.d/elpa/org-20160822/ox-beamer hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-beamer
/home/pok/.emacs.d/elpa/org-20160822/ox-odt hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-odt
/home/pok/.emacs.d/elpa/org-20160822/ob-core hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-core
/home/pok/.emacs.d/elpa/org-20160822/org-inlinetask hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-inlinetask
/home/pok/.emacs.d/elpa/org-20160822/org-rmail hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-rmail
/home/pok/.emacs.d/elpa/org-20160822/ox-icalendar hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-icalendar
/home/pok/.emacs.d/elpa/org-20160822/ob-lilypond hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-lilypond
/home/pok/.emacs.d/elpa/org-20160822/ob-picolisp hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-picolisp
/home/pok/.emacs.d/elpa/org-20160822/ob-sass hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-sass
/home/pok/.emacs.d/elpa/org-20160822/org-eshell hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-eshell
/home/pok/.emacs.d/elpa/org-20160822/ob-awk hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-awk
/home/pok/.emacs.d/elpa/org-20160822/ob-ruby hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-ruby
/home/pok/.emacs.d/elpa/org-20160822/ob-ocaml hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-ocaml
/home/pok/.emacs.d/elpa/org-20160822/ob-js hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-js
/home/pok/.emacs.d/elpa/org-20160822/org-footnote hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-footnote
/home/pok/.emacs.d/elpa/org-20160822/org-bbdb hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-bbdb
/usr/share/emacs24/site-lisp/latex-cjk-thai/thai-word hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/language/thai-word

Features:
(shadow emacsbug gnus-dup gnus-draft nndoc cus-edit ess-omg-l
ess-toolbar ess-mouse mouseme ess-menu ess-swv ess-noweb essd-els
ess-sas-d ess-sas-l ess-sas-a ess-sta-d ess-sta-l make-regexp ess-sp6-d
ess-dde ess-sp3-d ess-julia julia-mode ess-r-d ess-r-syntax
ess-r-completion ess-roxy essddr hideshow ess-help reporter
ess-r-package ess-s-l ess-site ess ess-inf ess-tracebug ess-mode
ess-noweb-mode ess-custom executable ess-generics ess-utils ess-bugs-l
ess-compat ess-lsp-l map esh-var esh-io esh-cmd esh-opt esh-ext esh-proc
esh-arg esh-groups eshell esh-module esh-mode esh-util dired-x autoload
tar-mode ffap php-mode flymake cc-mode cc-fonts cc-guess cc-menus
cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs hippie-exp vc
vc-dispatcher url-queue grep pulse vc-cvs slime-indentation
slime-cl-indent cl-indent slime-fancy slime-trace-dialog
slime-fontifying-fu slime-package-fu slime-references
slime-compiler-notes-tree slime-scratch slime-presentations bridge
slime-macrostep macrostep slime-mdot-fu slime-enclosing-context
slime-fuzzy slime-fancy-trace slime-fancy-inspector slime-c-p-c
slime-editing-commands slime-autodoc slime-repl elp slime-parse slime
lisp-mnt gud apropos etags xref project arc-mode archive-mode hyperspec
csv-mode magit-blame magit-stash magit-bisect magit-remote magit-commit
magit-sequence magit magit-apply magit-wip magit-log magit-diff
smerge-mode magit-core magit-autorevert autorevert filenotify
magit-process magit-popup magit-mode magit-git magit-section magit-utils
git-commit log-edit pcvs-util with-editor async-bytecomp async tramp-sh
tramp tramp-compat tramp-loaddefs trampver ucs-normalize shell dash
eieio-opt speedbar sb-image ezimage dframe cal-china lunar solar cal-dst
cal-bahai cal-islam cal-hebrew holidays hol-loaddefs cal-iso bookmark pp
tabify org-rmail org-mhe org-irc org-info org-gnus org-docview doc-view
jka-compr image-mode org-bbdb org-w3m org-table org-archive org-clock
org-indent ob-ditaa ob-plantuml org-bibtex org-element avl-tree bibtex
org-colview org-crypt org-habit org-agenda help-fns radix-tree misearch
multi-isearch thingatpt debbugs-gnu add-log debbugs soap-client warnings
rng-xsd rng-dt rng-util xsd-regexp debug cus-start cus-load mailalias
bbdb-message footnote ecomplete flow-fill vc-git diff-mode sort
gnus-cite shr-color color shr svg dom browse-url mail-extr gnus-async
gnus-bcklg qp gnus-ml disp-table gnus-topic mm-archive url-http url-gw
url-cache url-auth pop3 nnrss xml mm-url url url-proxy url-privacy
url-expand url-methods url-history url-cookie url-domsuf url-util
nndraft nnmh nnml utf-7 bbdb-gnus epa-file network-stream nsm starttls
gnus-agent gnus-srvr gnus-score score-mode nnvirtual gnus-msg nntp
gnus-cache pok-gnus gnus-icalendar org-capture gnus-art mm-uu mml2015
mm-view mml-smime smime dig mailcap icalendar diary-lib diary-loaddefs
nnir gnus-sum gnus-group gnus-undo bbdb-mua bbdb-com crm bbdb bbdb-site
timezone gnus-start gnus-cloud nnimap nnmail mail-source tls gnutls utf7
netrc nnoo parse-time gnus-spec gnus-int gnus-range message sendmail
puny dired dired-loaddefs rfc822 mml mml-sec epa derived epg mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
gnus-win gnus nnheader subr-x gnus-util rmail rmail-loaddefs rfc2047
rfc2045 ietf-drums mail-utils wid-edit linum paren paredit pok-init
bugz-mode yasnippet org advice org-macro org-footnote org-pcomplete
pcomplete org-list org-faces org-entities time-date noutline outline
org-version ob-emacs-lisp ob ob-tangle ob-ref ob-lob ob-table ob-exp
org-src ob-keys ob-comint ob-core ob-eval org-compat org-macs
org-loaddefs format-spec find-func cal-menu calendar cal-loaddefs gedcom
slime-autoloads reftex reftex-loaddefs reftex-vars edmacro kmacro ido
server filladapt dmacro mm-util mail-prsvr cl compile comint ansi-color
ring finder-inf tex-site info package epg-config url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
password-cache url-vars seq byte-opt gv bytecomp byte-compile cl-extra
help-mode easymenu cconv cl-loaddefs pcase cl-lib emacs-goodies-el
emacs-goodies-custom emacs-goodies-loaddefs easy-mmode mule-util tooltip
eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel
term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core term/tty-colors frame
cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote dbusbind inotify
dynamic-setting system-font-setting font-render-setting x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 2485829 182861)
 (symbols 48 72236 91)
 (miscs 40 3269 4672)
 (strings 32 298119 63000)
 (string-bytes 1 12887713)
 (vectors 16 103720)
 (vector-slots 8 2448384 49706)
 (floats 8 4985 2034)
 (intervals 56 225770 627)
 (buffers 976 190)
 (heap 1024 299121 -1842968))

[-- Attachment #2: calendar-event-problem.txt --]
[-- Type: application/octet-stream, Size: 41457 bytes --]

DESCRIPTION;LANGUAGE=en-US:Nn Nnnnn\,\n\nNnnnnnnnn nnn nnn nnnnnn nn nnnnnn
 nnnnnnn nnnn nnnnnnnnn. N nnnn nnnnnnnnn nn nnn nnnnnnnn nnnnnnn nnn nn nn
 nn nn-nnnnnnn nn Nnnnnnn nn 99.99 NNNN\n\nNnnn nnnnnnn\,\n\nNnnnn Nnnnnnnn
  | Nnnnnnnnnn Nnnnnnnn Nnnnn Nnnnnn\n\nNNN Nnnnnn Nnnnnnn\nNnnn Nnnnn Nnnn
 n\,\n99 Nnnnnnnnnn Nnnnnn\,\nNnnnnn\, NN9N 9NN\n+999999999999 | Nnnn: nnnn
 nnnnn@nnn.nnn\nnnnnnnnnn@nnn.nnn | nnn.nnn.nnn\n\n\n\n\n__________________
 ___________________________\nNnnn: Nnnnn N. Nnnnnnnnnnn [nnnnnn:nnn@nnnnnn
 nn.nn]\nNnnn: 99 Nnnnnn 9999 9:99 NN\nNn: NNN Nnnnnn <NNNNnnnnn@nnn.nnn>\n
 Nn: Nnnn Nnnnn <nnnnnn@nnn.nnn>\; Nnnn Nnnnnnnn <nnnnnnnnn@nnn.nnn>\; +NNN
 N <NNNN@nnn.nnn>\; +NN Nnnnnn <Nnnnnnnn@nnn.nnn>\nNnnnnnn: Nn: [NNN] Nn: N
 nnnnnn nnnnnnn nn NNN\n\n\nNnnnn\, N nnnn'n nnn nnn nnnn nnnnn nnn.  99:99
  NNN nnn nnnnnn\, nnn 99:99 NNN nn nnnnnn nn nn n nnnnnn\, nnn N'n nnnnn n
 nnnnnnnn. :)\n\n--\nNnn/Nnnnnnn\nNnnnn N. Nnnnnnnnnnn\nNnnnnnnn Nnnn NN\n\
 nNn Nnn\, Nnn 99 9999 nn 99:99\, NNN Nnnnnn nnnnn:\n\n> Nn Nnnnn\,\n>\n> N
 nnn nnn'nn nnnnnn n nnnn nnnn.\n>\n> Nnnnn nn nn nnnnnnnn nn nnnn Nnnnnnnn
 nnnnn nnnn nnnn nn nnnn nn 9.99nn NNN nnnnn?\n>\n> Nnnn nnnnnnn\,\n>\n> Nn
 nnn Nnnnnnnn | Nnnnnnnnnn Nnnnnnnn Nnnnn Nnnnnn\n>\n> NNN Nnnnnn Nnnnnnn\n
 > Nnnn Nnnnn Nnnnn\,\n> 99 Nnnnnnnnnn Nnnnnn\,\n> Nnnnnn\, NN9N 9NN\n> +99
 9999999999 | Nnnn: nnnnnnnnn@nnn.nnn\n> nnnnnnnnn@nnn.nnn | nnn.nnn.nnn\n>
 \n> -----Nnnnnnnn Nnnnnnn-----\n> Nnnn: NNN Nnnnnn\n> Nnnn: 99 Nnnnnn 9999
  9:99 NN\n> Nn: 'Nnnnn N. Nnnnnnnnnnn' <nnn@nnnnnnnn.nn>\; NNN Nnnnnn\n> <
 NNNNnnnnn@nnn.nnn>\n> Nn: Nnnn Nnnnn <nnnnnn@nnn.nnn>\; Nnnn Nnnnnnnn <nnn
 nnnnnn@nnn.nnn>\;\n> +NNNN <NNNN@nnn.nnn>\; +NN Nnnnnn <Nnnnnnnn@nnn.nnn>\
 n> Nnnnnnn: NN: [NNN] Nn: Nnnnnnn nnnnnnn nn NNN\n>\n> NN Nnnnn\,\n>\n> Nn
 nnnnnnn nnnnn nnn nn. Nnnnnn 9nn NNN nnnnn nn nnnnn. N nnnn nnnn nnn.\n>\n
 > N’nn nnnnnnn nnn n nnnnnnn nnnnnn nnn nnnn nnnnnnnn nnnnnnn.\n>\n> Nnn
 n nnnnnnn\,\n>\n> Nnnnn Nnnnnnnn | Nnnnnnnnnn Nnnnnnnn Nnnnn Nnnnnn\n>\n> 
 NNN Nnnnnn Nnnnnnn\n> Nnnn Nnnnn Nnnnn\,\n> 99 Nnnnnnnnnn Nnnnnn\,\n> Nnnn
 nn\, NN9N 9NN\n> +999999999999 | Nnnn: nnnnnnnnn@nnn.nnn\n> nnnnnnnnn@nnn.
 nnn | nnn.nnn.nnn\n>\n> -----Nnnnnnnn Nnnnnnn-----\n> Nnnn: Nnnnn N. Nnnnn
 nnnnnn [nnnnnn:nnn@nnnnnnnn.nn]\n> Nnnn: 99 Nnnnnn 9999 9:99 NN\n> Nn: NNN
  Nnnnnn <NNNNnnnnn@nnn.nnn>\n> Nn: Nnnn Nnnnn <nnnnnn@nnn.nnn>\; Nnnn Nnnn
 nnnn <nnnnnnnnn@nnn.nnn>\;\n> +NNNN <NNNN@nnn.nnn>\; +NN Nnnnnn <Nnnnnnnn@
 nnn.nnn>\n> Nnnnnnn: Nn: [NNN] Nn: Nnnnnnn nnnnnnn nn NNN\n>\n> Nn\, Nnnnn
 .\n>\n> Nnnnnn nnnnnnn Nnnnnnnnn nnnn nnnn nn nnnn nnn nn\, nnnnnnnnn nn n
 nnn\n> nnnn nn nnnn nnnnn nnn nnnn nnnnn.  Nn nn nnnn nn nnnn nnnnn NN nnn
 nn\,\n> nn nnn nn nn nnn nnnnn nnn nnnn?  Nnnnnnn nnnnn 99 NNN nnnnn nnn n
 n.\n>\n> Nnnnnn N nnnn nnn nn nnnn nnn nnnn nn?  Nn nnnnnn nnnn nn +99 999
 9 9999.\n>\n> --\n> Nnn/Nnnnnnn\n> Nnnnn N. Nnnnnnnnnnn\n> Nnnnnnnn Nnnn N
 N\n>\n> Nn Nnn\, Nnn 99 9999 nn 99:99\, NNN Nnnnnn nnnnn:\n>\n>> Nn Nnnnn\
 ,\n>>\n>> Nnnnn\, nnnn nnnnnnn nnn nnnn nnnnnnnnnnnn nn nnn nnnn. Nnnn nn 
 nnn nnnnnnn nnnn nnnnnn nn nnnn nn nnnnnn nnnnnnnnn nnn NNN nnnnnnnnn.\n>>
 \n>> Nn nnnnn nn nnnn nn nnnn nnnn nnnnnnnnn nnnn nnn nnnnnnnnnnnnn nnnnnn
 \n>> nnnnn N nnnn nnnnnnnn. N nnnnn nnnnnn nn nnnnn nnn nnnnnnnn nnnnn\,\n
 >> nnnn nnnn / nnnnnnnnnn nnnnnnnnn nnn nnnn nnnnnn nnnnnnnnn nnn nnn\n>> 
 nnnnnn / nnnnnnn / nnnnnn nnnnnnnnnnnnn\, nnnnnnnn nnnnn nnnnnn 99\n>> nnn
 nnnn nnnn n nnnnn nnnn. Nnnn nn nnnn nnnnn nnnnnn nnn nn nnn nnnnn\n>> nnn
 nnnn nnnn nnnnnnnnnn.\n>>\n>> Nnnnnn nnn nn nnnn nnnn nnnnnnnnnn nnnn / nn
 nnn nnn nnn.\n>>\n>> Nnnn nnnnnnn\,\n>>\n>> Nnnnn Nnnnnnnn | Nnnnnnnnnn Nn
 nnnnnn Nnnnn Nnnnnn\n>>\n>> NNN Nnnnnn Nnnnnnn\n>> Nnnn Nnnnn Nnnnn\,\n>> 
 99 Nnnnnnnnnn Nnnnnn\,\n>> Nnnnnn\, NN9N 9NN\n>> +999999999999 | Nnnn: nnn
 nnnnnn@nnn.nnn\n>> nnnnnnnnn@nnn.nnn | nnn.nnn.nnn\n>>\n>> -----Nnnnnnnn N
 nnnnnn-----\n>> Nnnn: Nnnnn N. Nnnnnnnnnnn [nnnnnn:nnn@nnnnnnnn.nn]\n>> Nn
 nn: 99 Nnnnnn 9999 9:99 NN\n>> Nn: NNN Nnnnnn <NNNNnnnnn@nnn.nnn>\n>> Nn: 
 Nnnn Nnnnn <nnnnnn@nnn.nnn>\; Nnnn Nnnnnnnn <nnnnnnnnn@nnn.nnn>\;\n>> +NNN
 N <NNNN@nnn.nnn>\; +NN Nnnnnn <Nnnnnnnn@nnn.nnn>\n>> Nnnnnnn: Nn: [NNN] Nn
 : Nnnnnnn nnnnnnn nn NNN\n>>\n>> Nnnnn nnnnn.\n>>\n>> N nnnn nnn nnnnnn nn
 nn nnnn\, nnn N'n nnnnnn nnnnn nnnn nnnnnn nn nn\n>> nnn.  N nnnnn nn nnn 
 nnnnn nnnnn nn nnn nnnnn nnnnnnnnn nnnn-nnn-nnnn\n>> nnn nnnn-nn-nnnn nnnn
 nn.  Nnn nn nnnnnnn nn nnn nnnnn nnnnn nnnnn?\n>> Nnn nnnn-nnn-nnnn\, nnn 
 nnnnnnnn nnnnnn nnnnnnn nn n nnnnnn\, nnnnn N\n>> nnn nnnnnnnnn n nnnnnn. 
  Nn nnnn nnnnnnn nn nnn nnn nnnnnnnnnn nnn\n>> nnnn nnnn nn nnnnn\, nn nnn
 nnnn nnnn nn nnn nnnn nnnnnn nnnnn?\n>>\n>> Nnnnnn nnn nn nnnn nn nn\, nn 
 nnnn nnnnnnnn nnn nnnn nnnnn nnnnn nn\n>> nnnnnn nn nnnnnnnnnn nnn nnn nnn
 nnnnnnn nnnnnn nnnn nn nnn nnnnn\n>> nnnn\, nnn nn nnnnn nn nnnn nn nnnn n
 nn nnnnnnn nnn nnnnnnnnnn.\n>>\n>> N nnnn nnnnnnnnnnnn nnnnnn nnnnnn nnnnn
 n\, nnnnn nnnnnn\, nnnnnnnnn\n>> nnnnn nnnnnnn nnn nnnnnn nnnnnnn\, NNN nn
 nnnn\, nnnnnnnnn nnnnnnnn\n>> nnnnnnnnnn\, nnn nnnnn nnnnnnnnnnnnn.\n>>\n>
 > Nnn nn nn nnnnnnn nn nnnnnnnnnn?  Nn nn nnnn nn nnnn nnnn nnnnnn nnnn nn
 nnnnnnn nn nnn?\n>>\n>> --\n>> Nnn/Nnnnnnn\n>> Nnnnn N. Nnnnnnnnnnn\n>> Nn
 nnnnnn Nnnn NN\n>>\n>> Nn Nnn\, Nnn 99 9999 nn 99:99\, NNN Nnnnnn nnnnn:\n
 >>\n>>> Nn Nnnnn\,\n>>>\n>>> Nnnnn - N nnnn nnnn nnnn nnnnn nnnnnnnnnn nn 
 nnn nnn nnnnnn nn nnn nnn nnnnnnnnnnn.\n>>>\n>>> Nnnnnn nnnnnnnn nnnnnnn.\
 n>>>\n>>> Nnnn nnnnnnn\,\n>>>\n>>> Nnnnn Nnnnnnnn | Nnnnnnnnnn Nnnnnnnn Nn
 nnn Nnnnnn\n>>>\n>>> NNN Nnnnnn Nnnnnnn\n>>> Nnnn Nnnnn Nnnnn\,\n>>> 99 Nn
 nnnnnnnn Nnnnnn\,\n>>> Nnnnnn\, NN9N 9NN\n>>> +999999999999 | Nnnn: nnnnnn
 nnn@nnn.nnn\n>>> nnnnnnnnn@nnn.nnn | nnn.nnn.nnn\n>>>\n>>> -----Nnnnnnnn N
 nnnnnn-----\n>>> Nnnn: Nnnnn N. Nnnnnnnnnnn [nnnnnn:nnn@nnnnnnnn.nn]\n>>> 
 Nnnn: 99 Nnnn 9999 9:99 NN\n>>> Nn: NNN Nnnnnn <NNNNnnnnn@nnn.nnn>\n>>> Nn
 : Nnnn Nnnnn <nnnnnn@nnn.nnn>\; Nnnn Nnnnnnnn <nnnnnnnnn@nnn.nnn>\;\n>>> +
 NNNN <NNNN@nnn.nnn>\; +NN Nnnnnn <Nnnnnnnn@nnn.nnn>\n>>> Nnnnnnn: Nn: [NNN
 ] Nn: Nnnnnnn nnnnnnn nn NNN\n>>>\n>>> Nn\, Nnnnn\n>>>\n>>> Nn nnnn nnnnnn
 nnn nnnn nnnn NN nnnn.  Nnnnn nnn nnnn n nnn nnnnn\n>>> nnnn nnnnn nnnnnnn
 n NN nnnn nnnnnnn nnn nn nnnn\, nn nn nnnnn nn nnnn\n>>> nn nnnn nnnn nnnn
 nn nnnn\, nnn nn'n nnn n nnnnnnnn.\n>>>\n>>> N nnn'n nnnnnnnn nnnn nn nnn 
 nnnn nnnnnnn nnn nnn nn nnnn\, nnn nnnnnnnnnn nn nnn nnnnn nnn nnnn NNN nn
 nnnnnnnnnnn nn nn nnn nnnnn nnn.\n>>>\n>>> --\n>>> Nnn/Nnnnnnn\n>>> Nnnnn 
 N. Nnnnnnnnnnn\n>>> Nnnnnnnn Nnnn NN\n>>>\n>>> Nn Nnn\, Nnn 99 9999 nn 99:
 99\, NNN Nnnnnn nnnnn:\n>>>\n>>>> Nn Nnnnn\,\n>>>>\n>>>> Nnnn nnn'nn nnn n
  nnnn nnnnnnn.\n>>>>\n>>>> Nn nn nn nnnnnnn nnnnnnnnn\, nnnnn N nnn nnnnnn
 n nn nnnn nnnnnn\n>>>> nnnnnnn. Nnn N nnnnnnn nnnnnnn nnn nnnn nn nnnnnnn 
 NN nnn NN nnnn\,\n>>>> nn nnn nn nnn nnnnn?\n>>>>\n>>>> Nnnnn nn nnnn nnnn
 nn\, N nnnn nn nnnnn nnn nnnn nnnnn nnnn nnnnnnnnn nn nnnn nnn nnnnnnnn NN
 N nnnnnnnnn nnn nn nnn nn nnnn nnnnn.\n>>>>\n>>>> Nnnn nnnnnnn\,\n>>>>\n>>
 >> Nnnnn Nnnnnnnn | Nnnnnnnnnn Nnnnnnnn Nnnnn Nnnnnn\n>>>>\n>>>> NNN Nnnnn
 n Nnnnnnn\n>>>> Nnnn Nnnnn Nnnnn\,\n>>>> 99 Nnnnnnnnnn Nnnnnn\,\n>>>> Nnnn
 nn\, NN9N 9NN\n>>>> +999999999999 | Nnnn: nnnnnnnnn@nnn.nnn\n>>>> nnnnnnnn
 n@nnn.nnn | nnn.nnn.nnn\n>>>>\n>>>> -----Nnnnnnnn Nnnnnnn-----\n>>>> Nnnn:
  Nnnnn N. Nnnnnnnnnnn [nnnnnn:nnn@nnnnnnnn.nn]\n>>>> Nnnn: 99 Nnnn 9999 9:
 99 NN\n>>>> Nn: NNN Nnnnnn <NNNNnnnnn@nnn.nnn>\n>>>> Nn: Nnnn Nnnnn <nnnnn
 n@nnn.nnn>\; Nnnn Nnnnnnnn <nnnnnnnnn@nnn.nnn>\;\n>>>> +NNNN <NNNN@nnn.nnn
 >\; +NN Nnnnnn <Nnnnnnnn@nnn.nnn>\n>>>> Nnnnnnn: Nn: [NNN] Nn: Nnnnnnn nnn
 nnnn nn NNN\n>>>>\n>>>> Nn\, Nnnnn.\n>>>>\n>>>> Nnnn nnnn nnnnnnnnn :)\n>>
 >>\n>>>> Nn nnn nnnnnn nnnnnnnnn-nn-nnnnnn nnn/nn nnnn-nnn-nnnn nnnnnn?  N
 \n>>>> nnnnn nnnn nn nnnn\, nnn nnn n nnnnnn nnnnnnnn nnnnn nnn nnnnnnnnn 
 -\n>>>> nnnn nnnn nnnnnnnn nnnnnnn nnn nnnnn nnnn nnn nnnnnnnnnnnn\, nn\n>
 >>> nnnnnnn nnnn nnnnn nnn nn nnnnnn?\n>>>>\n>>>> --\n>>>> Nnn/Nnnnnnn\n>>
 >> Nnnnn N. Nnnnnnnnnnn\n>>>> Nnnnnnnn Nnnn NN\n>>>>\n>>>> Nn Nnn\, Nnn 99
  9999 nn 99:99\, NNN Nnnnnn nnnnn:\n>>>>\n>>>>> Nn Nnnnn\,\n>>>>>\n>>>>> N
 nnnnn nnn nnnn nnnnn. N nnnnnn nnnn nn nnnnnn nn nnnnnnnn nn nnn nnnn nnnn
 nnnnnnn nn nnn nnnnnn nn.\n>>>>>\n>>>>> Nnn nn nnnnn nnnn Nnn\, nnn nn nnn
  nnnnn nn nn Nnnnnn.\n>>>>>\n>>>>> Nnnn n nnnnn nnnnnnn.\n>>>>>\n>>>>> Nnn
 nn Nnnnnnnn | Nnnnnnnnnn Nnnnnnnn Nnnnn Nnnnnn\n>>>>>\n>>>>> NNN Nnnnnn Nn
 nnnnn\n>>>>> Nnnn Nnnnn Nnnnn\,\n>>>>> 99 Nnnnnnnnnn Nnnnnn\,\n>>>>> Nnnnn
 n\, NN9N 9NN\n>>>>> +999999999999 | Nnnn: nnnnnnnnn@nnn.nnn\n>>>>> nnnnnnn
 nn@nnn.nnn | nnn.nnn.nnn\n>>>>>\n>>>>> Nnnnn\n>>>>>\n>>>>> -----Nnnnnnnn N
 nnnnnn-----\n>>>>> Nnnn: Nnnnn N. Nnnnnnnnnnn [nnnnnn:nnn@nnnnnnnn.nn]\n>>
 >>> Nnnn: 99 Nnnn 9999 9:99 NN\n>>>>> Nn: NNN Nnnnnn <NNNNnnnnn@nnn.nnn>\n
 >>>>> Nn: Nnnn Nnnnn <nnnnnn@nnn.nnn>\; Nnnn Nnnnnnnn\n>>>>> <nnnnnnnnn@nn
 n.nnn>\;\n>>>>> +NNNN <NNNN@nnn.nnn>\; +NN Nnnnnn <Nnnnnnnn@nnn.nnn>\n>>>>
 > Nnnnnnn: Nn: [NNN] Nn: Nnnnnnn nnnnnnn nn NNN\n>>>>>\n>>>>> Nn\, Nnnnn.\
 n>>>>>\n>>>>> Nn nnnnnn nnnnn nnn nnn nnn nn 9999 NN.  N nnnnnnnn nnn nnn\
 n>>>>> nnnnnnnn nnnnnnn nnnnn\, nnn nnn nnnnnn nn nnn nnnnnnnnn nnnnnn\n>>
 >>> nnnnnnn.  Nn nnn\n>>>>> NnnNnn(99) nnn nn 9.  Nnnnn nnnn nn nnnnn nnn 
 nnnn nnnnn nnnn nnn\n>>>>> nnnnnnn nnnnnnnn nnnnn\, N nnnnn nnnnnn NnnNnn 
 nn nnnnnn nn 9999\,\n>>>>> nn nn nnn nn nnn nnnnnnnn nnnnnnn.\n>>>>>\n>>>>
 > Nn nnnn nn nnnnnnnn nn nnn nnnnnnn nnnnnnnnnnn\, nn nnnn nnnn nnnnnnnnn 
 nn nnn nnnn nn nnnnnnnnnn?\n>>>>>\n>>>>> (Nnnn n nnnn nnnnnnn\, N'n nnnnnn
 n nnn nnn\, nnnn nnnnnnnn nnnn nn\n>>>>> nnnnnn.)\n>>>>> --\n>>>>> Nnn/Nnn
 nnnn\n>>>>> Nnnnn N. Nnnnnnnnnnn\n>>>>> Nnnnnnnn Nnnn NN\n>>>>>\n>>>>> Nn 
 Nnn\, Nnn 99 9999 nn 99:99\, NNN Nnnnnn nnnnn:\n>>>>>\n>>>>>> Nn Nnnnn\,\n
 >>>>>>\n>>>>>> N nnn nnn nnnnn nnnnnnnn. N nnnn nnnnnnnnn nnnn nnn nnnnnnn
 nnnn nnnn nnnn nn nnnn nnn nnnnnnnnn nnnnnnnnn nnnnnnnnnn nn nnnnn nnn nnn
 nnnnnn nnnnnnn:\n>>>>>>\n>>>>>> 9 nn 99 === nnnnnnnn\n>>>>>> 999 nn 999 ==
 = nnnn nnnnnnnn\n>>>>>> 999 nn 9999 === nnnnnnn nnnnnnn 999 nnn nnnn nnnnn
 nn\n>>>>>> 9999 nn 9999 === nnnnnnn nnnnnnn 999 nnn nnnnnn nnnnnnn\n>>>>>>
  9999 nn 9999 === nnnnnnn nnnnnnn 999\, 999 \, 999 \, 999 nnn nnnn\n>>>>>>
  nnnnnnn\n>>>>>> 9999 nn 9999 === nnnnnnn nnnnnnn 999\, 999\, 999\, 999 nn
 n nnnnnn\n>>>>>> nnnnnnn Nnnn 9999 === nnnnn nnnnn.\n>>>>>>\n>>>>>> Nn nnn
  nnnnn nnnn nn nnnnnn nnn nnnnnnnn nn nnnnn nnn nnnnnnnnnn nnnnn nnn nnnnn
 n nnn nnn nnnnnnnnn nnnnnnnnn nn nnnnnnnnn.\n>>>>>>\n>>>>>> Nnnn nnnnnnn\,
 \n>>>>>>\n>>>>>> Nnnnn Nnnnnnnn | Nnnnnnnnnn Nnnnnnnn Nnnnn Nnnnnn\n>>>>>>
 \n>>>>>> NNN Nnnnnn Nnnnnnn\n>>>>>> Nnnn Nnnnn Nnnnn\,\n>>>>>> 99 Nnnnnnnn
 nn Nnnnnn\,\n>>>>>> Nnnnnn\, NN9N 9NN\n>>>>>> +999999999999 | Nnnn: nnnnnn
 nnn@nnn.nnn\n>>>>>> nnnnnnnnn@nnn.nnn | nnn.nnn.nnn\n>>>>>>\n>>>>>> -----N
 nnnnnnn Nnnnnnn-----\n>>>>>> Nnnn: Nnnnn N. Nnnnnnnnnnn [nnnnnn:nnn@nnnnnn
 nn.nn]\n>>>>>> Nnnn: 99 Nnnn 9999 9:99 NN\n>>>>>> Nn: NNN Nnnnnn <NNNNnnnn
 n@nnn.nnn>\n>>>>>> Nn: Nnnn Nnnnn <nnnnnn@nnn.nnn>\; Nnnn Nnnnnnnn\n>>>>>>
  <nnnnnnnnn@nnn.nnn>\;\n>>>>>> +NNNN <NNNN@nnn.nnn>\; +NN Nnnnnn <Nnnnnnnn
 @nnn.nnn>\n>>>>>> Nnnnnnn: Nn: [NNN] Nn: Nnnnnnn nnnnnnn nn NNN\n>>>>>>\n>
 >>>>> Nnnnn nnnnn\, Nnnnn.\n>>>>>>\n>>>>>> N nnnn nnnnn nnnnnnnn n nnnn nn
 nnn nnn nnn nn 99 Nnnn Nnnn.  Nnn\n>>>>>> nnnnn nnn nnnnnnnn nnnn nnn nnnn
  "NnnnnNnnnnn Nnnnnn : NNNNNN\n>>>>>> NNNN nnn nnnnnn nn nnn<".  Nnn nn nn
 nn?\n>>>>>>\n>>>>>> --\n>>>>>> Nnn/Nnnnnnn\n>>>>>> Nnnnn N. Nnnnnnnnnnn\n>
 >>>>> Nnnnnnnn Nnnn NN\n>>>>>>\n>>>>>> Nn Nnn\, Nnn 99 9999 nn 99:99\, NNN
  Nnnnnn nnnnn:\n>>>>>>\n>>>>>>> Nn Nnnnn\,\n>>>>>>>\n>>>>>>> Nnnn nnn nnn 
 n nnnn nnnn nnn. Nnnn nn nnnn nn nn nnn nnnnnnn nnnnnnn nnnnnnNnnNN.\n>>>>
 >>>\n>>>>>>> Nnnnnn nnn nn nnnn nn N nnn nnnnnn nnnnnn nnnnnnn.\n>>>>>>>\n
 >>>>>>> Nnnn nnnnnnn\,\n>>>>>>>\n>>>>>>> Nnnnn Nnnnnnnn | Nnnnnnnnnn Nnnnn
 nnn Nnnnn Nnnnnn\n>>>>>>>\n>>>>>>> NNN Nnnnnn Nnnnnnn\n>>>>>>> Nnnn Nnnnn 
 Nnnnn\,\n>>>>>>> 99 Nnnnnnnnnn Nnnnnn\,\n>>>>>>> Nnnnnn\, NN9N 9NN\n>>>>>>
 > +999999999999 | Nnnn: nnnnnnnnn@nnn.nnn\n>>>>>>> nnnnnnnnn@nnn.nnn | nnn
 .nnn.nnn\n>>>>>>>\n>>>>>>> -----Nnnnnnnn Nnnnnnn-----\n>>>>>>> Nnnn: Nnnnn
  N. Nnnnnnnnnnn [nnnnnn:nnn@nnnnnnnn.nn]\n>>>>>>> Nnnn: 99 Nnnn 9999 9:99 
 NN\n>>>>>>> Nn: NNN Nnnnnn <NNNNnnnnn@nnn.nnn>\n>>>>>>> Nn: Nnnn Nnnnn <nn
 nnnn@nnn.nnn>\; Nnnn Nnnnnnnn\n>>>>>>> <nnnnnnnnn@nnn.nnn>\;\n>>>>>>> +NNN
 N <NNNN@nnn.nnn>\; +NN Nnnnnn <Nnnnnnnn@nnn.nnn>\n>>>>>>> Nnnnnnn: Nn: [NN
 N] Nn: Nnnnnnn nnnnnnn nn NNN\n>>>>>>>\n>>>>>>> Nn.\n>>>>>>>\n>>>>>>> N'nn
  nnnn nn nnnnnnn nnn n nnnnnn nn nnnnn.\n>>>>>>>\n>>>>>>> N'n nnnnnnnnn nn
 nnnnnnnnnn nnnnnn nn nn nnn NNN nnnnnn.  Nnn nnnnn nnn\, nn N nnnnnnnnn\, 
 nn nnn nnnn nnn nnnnnnNnnNN nn nnnnn.\n>>>>>>>\n>>>>>>> N nnnn nnnnn nnnnn
 nn nnnnnn nnnnnnn.\n>>>>>>>\n>>>>>>> --\n>>>>>>> Nnn/Nnnnnnn\n>>>>>>> Nnnn
 n N. Nnnnnnnnnnn\n>>>>>>> Nnnnnnnn Nnnn NN\n>>>>>>>\n>>>>>>> Nn Nnn\, Nnn 
 99 9999 nn 99:99\, NNN Nnnnnn nnnnn:\n>>>>>>>\n>>>>>>>> Nn Nnnnn\,\n>>>>>>
 >>\n>>>>>>>> Nnnnn nnn nnnnnnn nn nnnnn nnnnn nnn nn nnnnnn.\n>>>>>>>>\n>>
 >>>>>> Nnnn nnnnnnn\,\n>>>>>>>>\n>>>>>>>> Nnnnn Nnnnnnnn | Nnnnnnnnnn Nnnn
 nnnn Nnnnn Nnnnnn\n>>>>>>>>\n>>>>>>>> NNN Nnnnnn Nnnnnnn\n>>>>>>>> Nnnn Nn
 nnn Nnnnn\,\n>>>>>>>> 99 Nnnnnnnnnn Nnnnnn\,\n>>>>>>>> Nnnnnn\, NN9N 9NN\n
 >>>>>>>> +999999999999 | Nnnn: nnnnnnnnn@nnn.nnn\n>>>>>>>> nnnnnnnnn@nnn.n
 nn | nnn.nnn.nnn\n>>>>>>>>\n>>>>>>>> -----Nnnnnnnn Nnnnnnn-----\n>>>>>>>> 
 Nnnn: Nnnnn N. Nnnnnnnnnnn [nnnnnn:nnn@nnnnnnnn.nn]\n>>>>>>>> Nnnn: 99 Nnn
 n 9999 9:99 NN\n>>>>>>>> Nn: NNN Nnnnnn <NNNNnnnnn@nnn.nnn>\n>>>>>>>> Nn: 
 Nnnn Nnnnn <nnnnnn@nnn.nnn>\; Nnnn Nnnnnnnn\n>>>>>>>> <nnnnnnnnn@nnn.nnn>\
 ;\n>>>>>>>> +NNNN <NNNN@nnn.nnn>\; +NN Nnnnnn <Nnnnnnnn@nnn.nnn>\n>>>>>>>>
  Nnnnnnn: Nn: [NNN] Nn: Nnnnnnn nnnnnnn nn NNN\n>>>>>>>>\n>>>>>>>> Nn nnnn
 n.\n>>>>>>>>\n>>>>>>>> N nnnn nnnnnnnnnnnn\, nnn nn nnnnnnnnn nnnnn nnnn n
 nn nnnnnn.\n>>>>>>>> Nn nnnnnnn nn nn nnnn nn nn nnn nnnnnn nnn nnnnnnnnnn
  nnnnnnnn\n>>>>>>>> nn nnnnnnnnn nn nnnnn nnnnnnn.  Nn nnnn nnnn nnnn nn n
 nnnnnnnnn\n>>>>>>>> nn nn nnn?\n>>>>>>>>\n>>>>>>>> N nnnnnn nn nn 999.999.
 99.999:9999 nn nnnnnnnnnnnnn 99:99 NNN\,\n>>>>>>>> nnnn nnnnNnn NNNN/NNN n
 nn nnnnnnNnnNn NNNN.  Nn nnnnnnn nnn\n>>>>>>>> nnnnnnNnnNn nnnn nn nn nnnn
  nn nnn nnnnn nnnnnnnn nnnnnnnn?\n>>>>>>>>\n>>>>>>>> --\n>>>>>>>> Nnn/Nnnn
 nnn\n>>>>>>>> Nnnnn N. Nnnnnnnnnnn\n>>>>>>>> Nnnnnnnn Nnnn NN\n>>>>>>>>\n>
 >>>>>>> Nn Nnn\, Nnn 99 9999 nn 99:99\, Nnnnn N. Nnnnnnnnnnn nnnnn:\n>>>>>
 >>>\n>>>>>>>>> Nn\, Nnnnn.\n>>>>>>>>>\n>>>>>>>>> Nnnnn nnn\, N'nn nnn nn n
 n nnn nnn nnnnn nnnnnnn\, nnnnnn nnnnnnnn.\n>>>>>>>>>\n>>>>>>>>> --\n>>>>>
 >>>> Nnn/Nnnnnnn\n>>>>>>>>> Nnnnn N. Nnnnnnnnnnn\n>>>>>>>>> Nnnnnnnn Nnnn 
 NN\n>>>>>>>>>\n>>>>>>>>> Nn Nnn\, Nnn 99 9999 nn 99:99\, NNN Nnnnnn nnnnn:
 \n>>>>>>>>>\n>>>>>>>>>> Nn Nnnnn\,\n>>>>>>>>>>\n>>>>>>>>>> N'nn nnnn nnnnn
 nn nnn NN nnn nnnnnn nnn nnnnnnn. Nnnnnn\n>>>>>>>>>> nnnnnn 999.999.99.999
 . (Nnnn 9999)\n>>>>>>>>>>\n>>>>>>>>>> Nnnn nnnnnnn\,\n>>>>>>>>>>\n>>>>>>>>
 >> Nnnnn Nnnnnnnn | Nnnnnnnnnn Nnnnnnnn Nnnnn Nnnnnn\n>>>>>>>>>>\n>>>>>>>>
 >> NNN Nnnnnn Nnnnnnn\n>>>>>>>>>> Nnnn Nnnnn Nnnnn\,\n>>>>>>>>>> 99 Nnnnnn
 nnnn Nnnnnn\,\n>>>>>>>>>> Nnnnnn\, NN9N 9NN\n>>>>>>>>>> +999999999999 | Nn
 nn: nnnnnnnnn@nnn.nnn\n>>>>>>>>>> nnnnnnnnn@nnn.nnn | nnn.nnn.nnn\n>>>>>>>
 >>>\n>>>>>>>>>> -----Nnnnnnnn Nnnnnnn-----\n>>>>>>>>>> Nnnn: Nnnnn N. Nnnn
 nnnnnnn [nnnnnn:nnn@nnnnnnnn.nn]\n>>>>>>>>>> Nnnn: 99 Nnnn 9999 99:99 NN\n
 >>>>>>>>>> Nn: Nnnn Nnnnn <nnnnnn@nnn.nnn>\n>>>>>>>>>> Nn: NNN Nnnnnn <NNN
 Nnnnnn@nnn.nnn>\; Nnnn Nnnnnnnn\n>>>>>>>>>> <nnnnnnnnn@nnn.nnn>\; +NNNN <N
 NNN@nnn.nnn>\; +NN Nnnnnn\n>>>>>>>>>> <Nnnnnnnn@nnn.nnn>\n>>>>>>>>>> Nnnnn
 nn: Nn: [NNN] Nn: Nnnnnnn nnnnnnn nn NNN\n>>>>>>>>>>\n>>>>>>>>>> Nnnnnnnnn
 n\, nnnn nn nnnnnnn nnnn nn nnnnn nnnn nnn.  Nnnnnn!\n>>>>>>>>>> --\n>>>>>
 >>>>> Nnn/Nnnnnnn\n>>>>>>>>>> Nnnnn N. Nnnnnnnnnnn\n>>>>>>>>>> Nnnnnnnn Nn
 nn NN\n>>>>>>>>>>\n>>>>>>>>>> Nn Nnn\, Nnn 99 9999 nn 99:99\, Nnnn Nnnnn n
 nnnn:\n>>>>>>>>>>\n>>>>>>>>>>> Nnn\, n nnnnnnnnn nnnnnn nnnn nnnnnn nn nnn
  nnnnnnnnnnnn nn nnn nnnnn nnnnnnnnnn nn nnn nnnnnn nnnn. Nn nnnn nnn nnn'
 n nnnnnn nn nn nnnnnn?\n>>>>>>>>>>>\n>>>>>>>>>>> Nnnnnnn\,\n>>>>>>>>>>>\n>
 >>>>>>>>>> Nnnn Nnnnn\n>>>>>>>>>>>\n>>>>>>>>>>> N: +99 (9)99 9999 9999 | N
 : +99 (9)99 9999 9999\n>>>>>>>>>>>\n>>>>>>>>>>> -----Nnnnnnnn Nnnnnnn-----
 \n>>>>>>>>>>> Nnnn: Nnnnn N. Nnnnnnnnnnn [nnnnnn:nnn@nnnnnnnn.nn]\n>>>>>>>
 >>>> Nnnn: Nnnnnnnn\, Nnnn 9\, 9999 99:99 NN\n>>>>>>>>>>> Nn: Nnnn Nnnnn\n
 >>>>>>>>>>> Nn: NNN Nnnnnn\; Nnnn Nnnnnnnn\; +NNNN\; +NN Nnnnnn\n>>>>>>>>>
 >> Nnnnnnn: Nn: [NNN] Nn: Nnnnnnn nnnnnnn nn NNN\n>>>>>>>>>>>\n>>>>>>>>>>>
  Nn\, Nnnn.\n>>>>>>>>>>>\n>>>>>>>>>>> Nnnnnnnnnnnn nnnnnn nnnnnn nn nnnn n
 n nnnnnn@nnnnnnnn.nn\, nnnnn nn n nnnnnnnnnnnn nnnn nnn nnnnn nnnn nnn.\n>
 >>>>>>>>>>\n>>>>>>>>>>> Nnnn nnn nnnn nnnnnn NNN nnnnnn nnnn nnnnnnnn nnnn
 nnnnn nnnn n nnnnnnnnn nnnnnn?\n>>>>>>>>>>>\n>>>>>>>>>>> --\n>>>>>>>>>>> N
 nn/Nnnnnnn\n>>>>>>>>>>> Nnnnn N. Nnnnnnnnnnn\n>>>>>>>>>>> Nnnnnnnn Nnnn NN
 \n>>>>>>>>>>>\n>>>>>>>>>>> Nn Nnn\, Nnn 99 9999 nn 99:99\, Nnnn Nnnnn nnnn
 n:\n>>>>>>>>>>>\n>>>>>>>>>>>> Nn Nnnnn\,\n>>>>>>>>>>>>\n>>>>>>>>>>>> Nn NN
 N nnnnnn\, N nnn nnnnnnn nnnn nn nnnn nnnnnn nnnn nnnn\n>>>>>>>>>>>> nnnnn
  nn n nnnnnnnnn nnnnnn. Nnn nnnnnnnnnnnn nnnn nn\n>>>>>>>>>>>> nnnnnnnnnnn
  nn nn nnnnn nnnnnnnnnnnn\, nnnnnn nnn nn nnnn\n>>>>>>>>>>>> nnn nnnnnn nn
  nnn nnnnnnnnnn nn nnnn nnnnn.\n>>>>>>>>>>>>\n>>>>>>>>>>>> Nnn nnnnnnn nnn
 nnnnn nn NNN/NNN nnnnnn nn nnn nnnn nnnn nnnn nn nnnnn.\n>>>>>>>>>>>>\n>>>
 >>>>>>>>> Nnnnnnn\,\n>>>>>>>>>>>>\n>>>>>>>>>>>> Nnnn Nnnnn\n>>>>>>>>>>>>\n
 >>>>>>>>>>>> N: +99 (9)99 9999 9999 | N: +99 (9)99 9999 9999\n>>>>>>>>>>>>
 \n>>>>>>>>>>>> -----Nnnnnnnn Nnnnnnn-----\n>>>>>>>>>>>> Nnnn: Nnnnn N. Nnn
 nnnnnnnn [nnnnnn:nnn@nnnnnnnn.nn]\n>>>>>>>>>>>> Nnnn: Nnnnnnnn\, Nnnn 9\, 
 9999 99:99 NN\n>>>>>>>>>>>> Nn: NNN Nnnnnn\n>>>>>>>>>>>> Nn: Nnnn Nnnnnnnn
 \; +NNNN\; +NN Nnnnnn\n>>>>>>>>>>>> Nnnnnnn: Nn: [NNN] Nn: Nnnnnnn nnnnnnn
  nn NNN\n>>>>>>>>>>>>\n>>>>>>>>>>>> Nn\, Nnnnn.\n>>>>>>>>>>>>\n>>>>>>>>>>>
 > Nn nnnnnnnnnnn nnnn nnnnnnnnn nnnn nnnn.nnnnnnnn.nn\, 99.99.999.99.\n>>>
 >>>>>>>>>\n>>>>>>>>>>>> --\n>>>>>>>>>>>> Nnn/Nnnnnnn\n>>>>>>>>>>>> Nnnnn N
 . Nnnnnnnnnnn\n>>>>>>>>>>>> Nnnnnnnn Nnnn NN\n>>>>>>>>>>>>\n>>>>>>>>>>>> N
 n Nnn\, Nnn 99 9999 nn 99:99\, NNN Nnnnnn nnnnn:\n>>>>>>>>>>>>\n>>>>>>>>>>
 >>> Nn Nnnnn\,\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> N nnnn nnnnnnnn nnn nnnnnnn n
 nnn nn nnnn nnnnnn nnn nn nnnn nnnnnnnnn nnnnn N nnnn nnnnnnnnnnn nn nnn:\
 n>>>>>>>>>>>>>\n>>>>>>>>>>>>> Nn nnnnn n nnnnnnn nnnnnnn nnn NNN/NNN nnnnn
 n nn nnnn nnnnnn?  Nnn nn nnnn nnnnnn nnnnnnnnnnnnn nnnnnnnnn nn nnnn nn n
 nnnnnnnn nnnnnnn?\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> Nnnnnn nnnn nn nnnnnnn nnn
 nn:\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> Nn nnnnn nnnnnn nnn nn nnnnnn NN 999.99.
 999.99 nnnn nnn\n>>>>>>>>>>>>> nnnnnnnn nnn nnnn NNN nnnnnnn. Nnnnn nnn nn
 nnnn nnnnnn nnnn NN nnn nnnn nn nnnnnnnnnn nnnn nn nn nnn nnnnnnnnnn nnn n
 n nnn nnnnnnnn?\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> Nnn nnnn NNN nnnnnnn nn nnnn
 nn 9.9. Nnn99 nn nnn nnnnnnnnn\n>>>>>>>>>>>>> nnn nn nnnnnnnnn nn nnnn nn 
 nn nnnnnnnn. Nnn9999 nn nnnnnnnnn nnn nnn nnnnnnnnnn\, nnnnnnn\, nn nnn nn
 nnnnn nn nnnn nnnnnn 9999=NNN.\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> Nn nn nnnnnnn
 n nn nnnnnnnn nnnnnnnnnn nnn NNN nn Nnn nn nnn nn nnn nnnnnnn nn nnn nnnnn
 .\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> Nnnnn nnn9 nnn nnnnnnnnnn nnnn nn nnnn nn 
 nnn nnn nnnnnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> Nnnn nnnnnnn\,\n>>>>>>>>>>>>>
 \n>>>>>>>>>>>>> Nnnnn Nnnnnnnn | Nnnnnnnnnn Nnnnnnnn Nnnnn Nnnnnn\n>>>>>>>
 >>>>>>\n>>>>>>>>>>>>> NNN Nnnnnn Nnnnnnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> Nnn
 n Nnnnn Nnnnn\,\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> 99 Nnnnnnnnnn Nnnnnn\,\n>>>>
 >>>>>>>>>\n>>>>>>>>>>>>> Nnnnnn\, NN9N 9NN\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> +
 999999999999 | Nnnn: nnnnnnnnn@nnn.nnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> nnnnn
 nnnn@nnn.nnn | nnn.nnn.nnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> -----Nnnnnnnn Nnn
 nnnn-----\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> Nnnn: Nnnnn N. Nnnnnnnnnnn [nnnnnn
 :nnn@nnnnnnnn.nn]\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> Nnnn: 99 Nnnn 9999 9:99 NN
 \n>>>>>>>>>>>>>\n>>>>>>>>>>>>> Nn: Nnnnn Nnnnnnnn <nnnnnnnnn@nnn.nnn>\n>>>
 >>>>>>>>>>\n>>>>>>>>>>>>> Nn: NNN Nnnnnn <NNNNnnnnn@nnn.nnn>\; Nnnn Nnnnnn
 nn\n>>>>>>>>>>>>> <nnnnnnnnn@nnn.nnn>\;\n>>>>>>>>>>>>> +NNNN <NNNN@nnn.nnn
 >\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> Nnnnnnn: [NNN] Nn: Nnnnnnn nnnnnnn nn NNN\
 n>>>>>>>>>>>>>\n>>>>>>>>>>>>> Nn nnnnn.\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> Nnnn
  n nnnnnnnnn nnnnn nn nnnn nnnn nnnnn nnnnnnnnnnnnnnn\, N nnnnn.\n>>>>>>>>
 >>>>> Nn nn nnnn nn nnnnnnn 99=NNNN\;9999=NNN\, nn nn nnn nnnnnnn nn n Nnn
 nnnNnnNN nnn Nnnnnnnn nnnnn nn n NNN nnnnn?  Nnn nnn nnnn NNN nnnnnnn nn n
 nnnn 9.9\, nnnnnnn?\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> Nn nnnnnnn nnn nnn nnnnn
 n nn nnn 99 nnn nnnnnnnnn\, nn nnnn nnnnnn nn nn.\n>>>>>>>>>>>>>\n>>>>>>>>
 >>>>> Nn nnn'n nnnnnnnnn nnnnnnn n NnnnNnNnnnn\, nnn nn nnnnnnnn nn n nnn 
 nnnnn.  Nnnn nn nnnnn nnnnnnn nn nnn nnnnnn nn nnn nnn nnnnnn?\n>>>>>>>>>>
 >>>\n>>>>>>>>>>>>> Nn nnn nn nnnnnnnnnn nn nnnnnnnn nn nnnnnnn nnnn nnnnn\
 ,\n>>>>>>>>>>>>> nnn nnnnnn nnn nnnnnn nn nn nnnnn NNN nnn/nn NNN nnnnnn\,
 \n>>>>>>>>>>>>> nn N nnnnnnn nn nnn nnnnnnnnn nnnnnnnnn NNN nn nnnnnnn nnn
  nnnnnn nnnn nnn\, nnn nnnnnn nn nnnn nnnnnnnn nn nnnnnnnnn nnnnnn.  Nn nn
 nnn n nnnnnnn nnnnnnn nnn NNN/NNN nnnnnn nn nnnn nnnnnn?  Nnn nn nnnn nnnn
 nn nnnnnnnnnnnnn nnnnnnnnn nn nnnn nn nnnnnnnnn nnnnnnn?\n>>>>>>>>>>>>>\n>
 >>>>>>>>>>>> Nn nnnn nnn nnn Nnnnnnn(9) nnn nnn nnnnn nn nnnnnnn nnn nnnnn
 nnnnn nnnnnnn.  Nnnn nnnn nnnnnnnn nn nnnn nn nnn nnn nnnnnn?\n>>>>>>>>>>>
 >>\n>>>>>>>>>>>>> --\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> Nnn/Nnnnnnn\n>>>>>>>>>>
 >>>\n>>>>>>>>>>>>> Nnnnn N. Nnnnnnnnnnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> Nnnn
 nnnn Nnnn NN\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> Nn Nnn\, Nnn 99 9999 nn 99:99\,
  Nnnnn N. Nnnnnnnnnnn nnnnn:\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>> Nn.\n>>>>>>>>>
 >>>>\n>>>>>>>>>>>>>>\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>> Nnnnn nnnnn nnn nnnnn\
 , N nnn nnn nn nnn nnnnnn nnn nnnn\n>>>>>>>>>>>>>> nn nnnn nnnn\,\n>>>>>>>
 >>>>>>\n>>>>>>>>>>>>>> nnn nnnnn nnnn nn nnnnnnnnn nnnnnnnn nn nn nnnnn nn
 nnnnnn.\n>>>>>>>>>>>>>> N nn nnnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>> nnnnnnnnn
  nnn nnnnnnnnnnnnn nnn nnnn nn\, nnn nnnn nnn\n>>>>>>>>>>>>>> nnnn nn nnn 
 nnnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>> nnn nnnnnnnnn.\n>>>>>>>>>>>>>\n>>>>>>>
 >>>>>>>\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>> N nnnn nnn nnnnnnnnn nnnnnnnn\, nnn
 nnnnnn nnn NNN nnnnnnn\n>>>>>>>>>>>>>> nnn nnnn nnn\n>>>>>>>>>>>>>\n>>>>>>
 >>>>>>>> nn: nnnnn NN\, nnn nnn nnn nnnnnnnnn nnnnnnnnnnn nnn NNN\n>>>>>>>
 >>>>>>> nn nnnn nnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>> Nnnnnnnn?\n>>>>>>>>>>>>
 >\n>>>>>>>>>>>>>>\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>> --\n>>>>>>>>>>>>>\n>>>>>>
 >>>>>>>> Nnn/Nnnnnnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>> Nnnnn N. Nnnnnnnnnnn\n
 >>>>>>>>>>>>>\n>>>>>>>>>>>>>> Nnnnnnnn Nnnn NN\n>>>>>>>>>>>>>\n>>>>>>>>>>>
 >>>\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>> Nn Nnn\, Nnn 99 9999 nn 99:99\, Nnnnn N
 nnnnnnn nnnnn:\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>\n>>>>>>>>>>>>>\n>>>>>>>>>>>>
 >>> Nn Nnnnn\,\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>>\n>>>>>>>>>>>>>\n>>>>>>>>>>>
 >>>> Nnnnnn nnn nn nnnn nn nnnnnnnn nnnnn nn nnnnnnn\, nnn N nnnn nn nnnnn
  nn nnnnnn. Nn nnn nnnnn nnnnnn nn nnnnnnn nnnn nnn nnnnn nnnnnn nnnn nnnn
 .\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>>\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> Nnnn nnn
 nnnn\,\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>>\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> Nnn
 nn Nnnnnnnn | Nnnnnnnnnn Nnnnnnnn Nnnnn Nnnnnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>
 >>>> [Nnnnnnnnnnn: Nnnnnnnnnnn:\n>>>>>>>>>>>>>>> nnn:nnnnn999.nnn@99NN9999
 .99999N99]\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> NNN Nnnnnn Nnnnnnn\n>>>>>>>>>>>
 >>\n>>>>>>>>>>>>>>> Nnnn Nnnnn Nnnnn\,\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> 99 
 Nnnnnnnnnn Nnnnnn\,\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> Nnnnnn\, NN9N 9NN\n>>>
 >>>>>>>>>>\n>>>>>>>>>>>>>>> +999999999999 | Nnnn:\n>>>>>>>>>>>>>>> +nnnnnn
 nnn@nnn.nnn<nnnnnn:nnnnnnnnn@nnn.nnn>\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> nnnn
 nnnnn@nnn.nnn<nnnnnn:nnnnnnnnn@nnn.nnn> |\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> 
 nnn.nnn.nnn<nnnn://nnn.nnn.nnn/>\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>>\n>>>>>>>>
 >>>>>\n>>>>>>>>>>>>>>> Nnnn: Nnnnn Nnnnnnnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>
 > Nnnn: 99 Nnn 9999 99:99 NN\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> Nn: 'nnn@nnnn
 nnnn.nn' <nnn@nnnnnnnn.nn>\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> Nn: +NNN Nnnnnn
  <NNNNnnnnn@nnn.nnn>\; Nnnn Nnnnnnnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> <nnnn
 nnnnn@nnn.nnn>\; +NNNN <NNNN@nnn.nnn>\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> Nnnn
 nnn: Nnnnnnn nnnnnnn nn NNN\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>>\n>>>>>>>>>>>>>
 \n>>>>>>>>>>>>>>> Nn Nnnnn\,\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>>\n>>>>>>>>>>>>
 >\n>>>>>>>>>>>>>>> N nnnn nnnn nnnnn nnnn nnnnnnn nnnnnnnnnnn nn nnnnnnn n
 n nnnnnnn nnnnnnn nnnnnnn nnnnnnn nnnnnnnnnn nnn NNN.\n>>>>>>>>>>>>>\n>>>>
 >>>>>>>>>>>\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> N nnnnnnnnnn nnnn nnn nnn nnnn
 nnn nn nnnnnn n nnnnnnnnn\n>>>>>>>>>>>>>>> NNN NNN\, nnn\n>>>>>>>>>>>>>\n>
 >>>>>>>>>>>>>> nn nnnn nnnn nn nnnn nnn nnnnnnn nnn nnnn nnn nnnnn. N\n>>>
 >>>>>>>>>>>> nnnnnnnnnn nnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> nnnn nn nnnnnn
 n NNN nnnnnn nnnn. N nnnn nnnnnnnn nnn\n>>>>>>>>>>>>>>> nnnnnnn NNN\n>>>>>
 >>>>>>>>\n>>>>>>>>>>>>>>> nnnnnnnnnnnnn\, nnnnn nnn nnn nnnn nnnnnn - Nnn 
 NNN\n>>>>>>>>>>>>>>> nnnnnn nnnn nnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> nnnn 
 nnnnnnnnn nn n NNN nnnnnnnn nnnnn\, nn nnnn nn nnn\n>>>>>>>>>>>>>>> NNN NN
 N nnnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> nn nnn nnn NNN nnnnnn nn nn nnnnnnn
 .\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>>\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> N nnn nn
  nnnnnnnnn nnnn n NNN nnnnnnn nnn nn nnn nnn nn nnnnn. Nn nnn nnnn nnnn nn
 nn nnnnnnn\, nn nnnnn n nnn NNN nnnnnnn nnn nnn\, nn nnn nnnnnn.\n>>>>>>>>
 >>>>>\n>>>>>>>>>>>>>>>\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> nnnnnnNnnnNN=NNNN\n
 >>>>>>>>>>>>>\n>>>>>>>>>>>>>>> NnnnnnNnnnNN=NNN\n>>>>>>>>>>>>>\n>>>>>>>>>>
 >>>>> Nnnn=9999\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>>\n>>>>>>>>>>>>>\n>>>>>>>>>>
 >>>>> N nnnn nnnnnnn nn nnnnnnn nnnn nnn.\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>>\
 n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> Nnnn nnnnnnn\,\n>>>>>>>>>>>>>\n>>>>>>>>>>>
 >>>>\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> Nnnnn Nnnnnnnn | Nnnnnnnnnn Nnnnnnnn 
 Nnnnn Nnnnnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> [Nnnnnnnnnnn: Nnnnnnnnnnn:\n>
 >>>>>>>>>>>>>> nnn:nnnnn999.nnn@99NN9999.99999N99]\n>>>>>>>>>>>>>\n>>>>>>>
 >>>>>>>> NNN Nnnnnn Nnnnnnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> Nnnn Nnnnn Nnn
 nn\,\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> 99 Nnnnnnnnnn Nnnnnn\,\n>>>>>>>>>>>>>
 \n>>>>>>>>>>>>>>> Nnnnnn\, NN9N 9NN\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> +99999
 9999999 | Nnnn:\n>>>>>>>>>>>>>>> +nnnnnnnnn@nnn.nnn<nnnnnn:nnnnnnnnn@nnn.n
 nn>\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> nnnnnnnnn@nnn.nnn<nnnnnn:nnnnnnnnn@nnn
 .nnn> |\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> nnn.nnn.nnn<nnnn://nnn.nnn.nnn/>\n
 >>>>>>>>>>>>>\n>>>>>>>>>>>>>>>\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> Nnnn n-nnnn
  nnn nnn nnnnnnnnnnn nnn nnnnnnnn nnnn nnn\n>>>>>>>>>>>>>>> nnn nnnnnnnnnn
 \n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> nn nnnnnn nn nnnn nn nn nnnnnnnnn nnn nnn
  nnnnnnn\n>>>>>>>>>>>>>>> nnnnnnnnnnn nnnn nn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>
 >>> nnnnnnnnnnnn\, nnnnnnnnnn\, nnnnnn nnnnnnnnnnn\, nn nnnnnnn\n>>>>>>>>>
 >>>>>> nn nnnnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> nnnnnnnnnnnn nn nnn nn nnn
 nnnnnnn. Nnn nnnnnnnnnnnn nnn\,\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> nnnnnnnnnn
 nnn nn nnnnnnn nn nnnn nnnnnnnnnnnn nn nnn\n>>>>>>>>>>>>>>> nnnnnnnnnnn nn
 \n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> nn nn nnnnnnnnnn nnn nnn nn nnnnnnnn. Nn 
 nnn nnnn\n>>>>>>>>>>>>>>> nnnnnnnn nnnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> nn
 nnnnnnnnnn nn nnnnn\, nnnnnn nnnnnn nnn nnnnnn\n>>>>>>>>>>>>>>> nnnnnnnnnn
 n nn nnnnnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> n-nnnn\, nnn nnnnnnnnnnn nnnnn
 n nn nnnnnnn nnnn n-nnnn\,\n>>>>>>>>>>>>>>> nnn\n>>>>>>>>>>>>>\n>>>>>>>>>>
 >>>>> nnnnnnnnnnn\, nnn nnn nnnnnn (nnnnnnn nn nnnnn). Nnnnnn\n>>>>>>>>>>>
 >>>> nnnnnnnnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> nnnnnn nn nnnn n-nnnn\, nnn
 nnnn nn nnnn nnnnnnn nnnnnn nn\n>>>>>>>>>>>>>>> nnnnnnnnn nn\n>>>>>>>>>>>>
 >\n>>>>>>>>>>>>>>> n nnnnnnn nn nnnnnnnnnn nnnnnnnnn. Nnn nnnnnnnnnn\n>>>>
 >>>>>>>>>>> nnnnnnnnn\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>> nnnnnnnnnnn nnn nnnn
 nnnnnnn nnnnnnnnn NNN'n nnnnnnnn nnn\n>>>>>>>>>>>>>>> nnnnnnnn\,\n>>>>>>>>
 >>>>>\n>>>>>>>>>>>>>>> nnnnnn nnnnn nn nnn nnnnnnnnn\n>>>>>>>>>>>>>\n>>>>>
 >>>>>>>>>> nnnn:\n>>>>>>>>>>>>>\n>>>>>>>>>>>>>>>\n>>>>>>>>>>>>>\n>>>>>>>>>
 >>>>>> nnnn://nnn.nnn.nnn/nnnnn/nnnnnn-nnnnnnnnnnn\n>>>>>>>>>>>>>\n>>>>>>>
 >>>>>> Nnnn n-nnnn nnn nnn nnnnnnnnnnn nnn nnnnnnnn nnnn nnn nnn\n>>>>>>>>
 >>>>> nnnnnnnnnn nn nnnnnn nn nnnn nn nn nnnnnnnnn nnn nnn\n>>>>>>>>>>>>> 
 nnnnnnn nnnnnnnnnnn nnnn nn nnnnnnnnnnnn\, nnnnnnnnnn\,\n>>>>>>>>>>>>> nnn
 nnn nnnnnnnnnnn\, nn nnnnnnn nn nnnnn nnnnnnnnnnnn nn\n>>>>>>>>>>>>> nnn n
 n nnnnnnnnnn. Nnn nnnnnnnnnnnn nnn\, nnnnnnnnnnnnn nn nnnnnnn nn nnnn nnnn
 nnnnnnnn nn nnn nnnnnnnnnnn nn nn nn nnnnnnnnnn nnn nnn nn nnnnnnnn. Nn nn
 n nnnn nnnnnnnn nnnn nnnnnnnnnnnn nn nnnnn\, nnnnnn nnnnnn nnn nnnnnn nnnn
 nnnnnnn nn nnnnnn n-nnnn\, nnn nnnnnnnnnnn nnnnnn nn nnnnnnn nnnn n-nnnn\,
  nnn nnnnnnnnnnn\, nnn nnn nnnnnn (nnnnnnn nn nnnnn). Nnnnnn nnnnnnnnn nnn
 nnn nn nnnn n-nnnn\, nnnnnnn nn nnnn nnnnnnn nnnnnn nn nnnnnnnnn nn n nnnn
 nnn nn nnnnnnnnnn nnnnnnnnn. Nnn nnnnnnnnnn nnnnnnnnn nnnnnnnnnnn nnn nnnn
 nnnnnnn nnnnnnnnn NNN’n nnnnnnnn nnn nnnnnnnn\, nnnnnn nnnnn nn nnn nnnn
 nnnnn nnnn:\n>>>>>>>>>>>>>\n>>>>>>>>>>>>> nnnn://nnn.nnn.nnn/nnnnn/nnnnnn-
 nnnnnnnnnnn\n>>>>>>>>>>>> Nnnn n-nnnn nnn nnn nnnnnnnnnnn nnn nnnnnnnn nnn
 n nnn nnn nnnnnnnnnn nn nnnnnn nn nnnn nn nn nnnnnnnnn nnn nnn nnnnnnn nnn
 nnnnnnnn nnnn nn nnnnnnnnnnnn\, nnnnnnnnnn\, nnnnnn nnnnnnnnnnn\, nn nnnnn
 nn nn nnnnn nnnnnnnnnnnn nn nnn nn nnnnnnnnnn. Nnn nnnnnnnnnnnn nnn\, nnnn
 nnnnnnnnn nn nnnnnnn nn nnnn nnnnnnnnnnnn nn nnn nnnnnnnnnnn nn nn nn nnnn
 nnnnnn nnn nnn nn nnnnnnnn. Nn nnn nnnn nnnnnnnn nnnn nnnnnnnnnnnn nn nnnn
 n\, nnnnnn nnnnnn nnn nnnnnn nnnnnnnnnnn nn nnnnnn n-nnnn\, nnn nnnnnnnnnn
 n nnnnnn nn nnnnnnn nnnn n-nnnn\, nnn nnnnnnnnnnn\, nnn nnn nnnnnn (nnnnnn
 n nn nnnnn). Nnnnnn nnnnnnnnn nnnnnn nn nnnn n-nnnn\, nnnnnnn nn nnnn nnnn
 nnn nnnnnn nn nnnnnnnnn nn n nnnnnnn nn nnnnnnnnnn nnnnnnnnn. Nnn nnnnnnnn
 nn nnnnnnnnn nnnnnnnnnnn nnn nnnnnnnnnnn nnnnnnnnn NNN’n nnnnnnnn nnn nn
 nnnnnn\, nnnnnn nnnnn nn nnn nnnnnnnnn nnnn:\n>>>>>>>>>>>>\n>>>>>>>>>>>> n
 nnn://nnn.nnn.nnn/nnnnn/nnnnnn-nnnnnnnnnnn\n>>>>>>>>>>> Nnnn n-nnnn nnn nn
 n nnnnnnnnnnn nnn nnnnnnnn nnnn nnn nnn nnnnnnnnnn nn nnnnnn nn nnnn nn nn
  nnnnnnnnn nnn nnn nnnnnnn nnnnnnnnnnn nnnn nn nnnnnnnnnnnn\, nnnnnnnnnn\,
  nnnnnn nnnnnnnnnnn\, nn nnnnnnn nn nnnnn nnnnnnnnnnnn nn nnn nn nnnnnnnnn
 n. Nnn nnnnnnnnnnnn nnn\, nnnnnnnnnnnnn nn nnnnnnn nn nnnn nnnnnnnnnnnn nn
  nnn nnnnnnnnnnn nn nn nn nnnnnnnnnn nnn nnn nn nnnnnnnn. Nn nnn nnnn nnnn
 nnnn nnnn nnnnnnnnnnnn nn nnnnn\, nnnnnn nnnnnn nnn nnnnnn nnnnnnnnnnn nn 
 nnnnnn n-nnnn\, nnn nnnnnnnnnnn nnnnnn nn nnnnnnn nnnn n-nnnn\, nnn nnnnnn
 nnnnn\, nnn nnn nnnnnn (nnnnnnn nn nnnnn). Nnnnnn nnnnnnnnn nnnnnn nn nnnn
  n-nnnn\, nnnnnnn nn nnnn nnnnnnn nnnnnn nn nnnnnnnnn nn n nnnnnnn nn nnnn
 nnnnnn nnnnnnnnn. Nnn nnnnnnnnnn nnnnnnnnn nnnnnnnnnnn nnn nnnnnnnnnnn nnn
 nnnnnn NNN’n nnnnnnnn nnn nnnnnnnn\, nnnnnn nnnnn nn nnn nnnnnnnnn nnnn:
 \n>>>>>>>>>>>\n>>>>>>>>>>> nnnn://nnn.nnn.nnn/nnnnn/nnnnnn-nnnnnnnnnnn\n>>
 >>>>>>>> Nnnn n-nnnn nnn nnn nnnnnnnnnnn nnn nnnnnnnn nnnn nnn nnn nnnnnnn
 nnn nn nnnnnn nn nnnn nn nn nnnnnnnnn nnn nnn nnnnnnn nnnnnnnnnnn nnnn nn 
 nnnnnnnnnnnn\, nnnnnnnnnn\, nnnnnn nnnnnnnnnnn\, nn nnnnnnn nn nnnnn nnnnn
 nnnnnnn nn nnn nn nnnnnnnnnn. Nnn nnnnnnnnnnnn nnn\, nnnnnnnnnnnnn nn nnnn
 nnn nn nnnn nnnnnnnnnnnn nn nnn nnnnnnnnnnn nn nn nn nnnnnnnnnn nnn nnn nn
  nnnnnnnn. Nn nnn nnnn nnnnnnnn nnnn nnnnnnnnnnnn nn nnnnn\, nnnnnn nnnnnn
  nnn nnnnnn nnnnnnnnnnn nn nnnnnn n-nnnn\, nnn nnnnnnnnnnn nnnnnn nn nnnnn
 nn nnnn n-nnnn\, nnn nnnnnnnnnnn\, nnn nnn nnnnnn (nnnnnnn nn nnnnn). Nnnn
 nn nnnnnnnnn nnnnnn nn nnnn n-nnnn\, nnnnnnn nn nnnn nnnnnnn nnnnnn nn nnn
 nnnnnn nn n nnnnnnn nn nnnnnnnnnn nnnnnnnnn. Nnn nnnnnnnnnn nnnnnnnnn nnnn
 nnnnnnn nnn nnnnnnnnnnn nnnnnnnnn NNN’n nnnnnnnn nnn nnnnnnnn\, nnnnnn n
 nnnn nn nnn nnnnnnnnn nnnn:\n>>>>>>>>>>\n>>>>>>>>>> nnnn://nnn.nnn.nnn/nnn
 nn/nnnnnn-nnnnnnnnnnn\n>>>>>>>> Nnnn n-nnnn nnn nnn nnnnnnnnnnn nnn nnnnnn
 nn nnnn nnn nnn nnnnnnnnnn nn nnnnnn nn nnnn nn nn nnnnnnnnn nnn nnn nnnnn
 nn nnnnnnnnnnn nnnn nn nnnnnnnnnnnn\, nnnnnnnnnn\, nnnnnn nnnnnnnnnnn\, nn
  nnnnnnn nn nnnnn nnnnnnnnnnnn nn nnn nn nnnnnnnnnn. Nnn nnnnnnnnnnnn nnn\
 , nnnnnnnnnnnnn nn nnnnnnn nn nnnn nnnnnnnnnnnn nn nnn nnnnnnnnnnn nn nn n
 n nnnnnnnnnn nnn nnn nn nnnnnnnn. Nn nnn nnnn nnnnnnnn nnnn nnnnnnnnnnnn n
 n nnnnn\, nnnnnn nnnnnn nnn nnnnnn nnnnnnnnnnn nn nnnnnn n-nnnn\, nnn nnnn
 nnnnnnn nnnnnn nn nnnnnnn nnnn n-nnnn\, nnn nnnnnnnnnnn\, nnn nnn nnnnnn (
 nnnnnnn nn nnnnn). Nnnnnn nnnnnnnnn nnnnnn nn nnnn n-nnnn\, nnnnnnn nn nnn
 n nnnnnnn nnnnnn nn nnnnnnnnn nn n nnnnnnn nn nnnnnnnnnn nnnnnnnnn. Nnn nn
 nnnnnnnn nnnnnnnnn nnnnnnnnnnn nnn nnnnnnnnnnn nnnnnnnnn NNN’n nnnnnnnn 
 nnn nnnnnnnn\, nnnnnn nnnnn nn nnn nnnnnnnnn nnnn:\n>>>>>>>>\n>>>>>>>> nnn
 n://nnn.nnn.nnn/nnnnn/nnnnnn-nnnnnnnnnnn\n>>>>>>> Nnnn n-nnnn nnn nnn nnnn
 nnnnnnn nnn nnnnnnnn nnnn nnn nnn nnnnnnnnnn nn nnnnnn nn nnnn nn nn nnnnn
 nnnn nnn nnn nnnnnnn nnnnnnnnnnn nnnn nn nnnnnnnnnnnn\, nnnnnnnnnn\, nnnnn
 n nnnnnnnnnnn\, nn nnnnnnn nn nnnnn nnnnnnnnnnnn nn nnn nn nnnnnnnnnn. Nnn
  nnnnnnnnnnnn nnn\, nnnnnnnnnnnnn nn nnnnnnn nn nnnn nnnnnnnnnnnn nn nnn n
 nnnnnnnnnn nn nn nn nnnnnnnnnn nnn nnn nn nnnnnnnn. Nn nnn nnnn nnnnnnnn n
 nnn nnnnnnnnnnnn nn nnnnn\, nnnnnn nnnnnn nnn nnnnnn nnnnnnnnnnn nn nnnnnn
  n-nnnn\, nnn nnnnnnnnnnn nnnnnn nn nnnnnnn nnnn n-nnnn\, nnn nnnnnnnnnnn\
 , nnn nnn nnnnnn (nnnnnnn nn nnnnn). Nnnnnn nnnnnnnnn nnnnnn nn nnnn n-nnn
 n\, nnnnnnn nn nnnn nnnnnnn nnnnnn nn nnnnnnnnn nn n nnnnnnn nn nnnnnnnnnn
  nnnnnnnnn. Nnn nnnnnnnnnn nnnnnnnnn nnnnnnnnnnn nnn nnnnnnnnnnn nnnnnnnnn
  NNN’n nnnnnnnn nnn nnnnnnnn\, nnnnnn nnnnn nn nnn nnnnnnnnn nnnn:\n>>>>
 >>>\n>>>>>>> nnnn://nnn.nnn.nnn/nnnnn/nnnnnn-nnnnnnnnnnn\n>>>>>>\n>>>>>> N
 nnn n-nnnn nnn nnn nnnnnnnnnnn nnn nnnnnnnn nnnn nnn nnn nnnnnnnnnn nn nnn
 nnn nn nnnn nn nn nnnnnnnnn nnn nnn nnnnnnn nnnnnnnnnnn nnnn nn nnnnnnnnnn
 nn\, nnnnnnnnnn\, nnnnnn nnnnnnnnnnn\, nn nnnnnnn nn nnnnn nnnnnnnnnnnn nn
  nnn nn nnnnnnnnnn. Nnn nnnnnnnnnnnn nnn\, nnnnnnnnnnnnn nn nnnnnnn nn nnn
 n nnnnnnnnnnnn nn nnn nnnnnnnnnnn nn nn nn nnnnnnnnnn nnn nnn nn nnnnnnnn.
  Nn nnn nnnn nnnnnnnn nnnn nnnnnnnnnnnn nn nnnnn\, nnnnnn nnnnnn nnn nnnnn
 n nnnnnnnnnnn nn nnnnnn n-nnnn\, nnn nnnnnnnnnnn nnnnnn nn nnnnnnn nnnn n-
 nnnn\, nnn nnnnnnnnnnn\, nnn nnn nnnnnn (nnnnnnn nn nnnnn). Nnnnnn nnnnnnn
 nn nnnnnn nn nnnn n-nnnn\, nnnnnnn nn nnnn nnnnnnn nnnnnn nn nnnnnnnnn nn 
 n nnnnnnn nn nnnnnnnnnn nnnnnnnnn. Nnn nnnnnnnnnn nnnnnnnnn nnnnnnnnnnn nn
 n nnnnnnnnnnn nnnnnnnnn NNN’n nnnnnnnn nnn nnnnnnnn\, nnnnnn nnnnn nn nn
 n nnnnnnnnn nnnn:\n>>>>>>\n>>>>>> nnnn://nnn.nnn.nnn/nnnnn/nnnnnn-nnnnnnnn
 nnn\n>>>>> Nnnn n-nnnn nnn nnn nnnnnnnnnnn nnn nnnnnnnn nnnn nnn nnn nnnnn
 nnnnn nn nnnnnn nn nnnn nn nn nnnnnnnnn nnn nnn nnnnnnn nnnnnnnnnnn nnnn n
 n nnnnnnnnnnnn\, nnnnnnnnnn\, nnnnnn nnnnnnnnnnn\, nn nnnnnnn nn nnnnn nnn
 nnnnnnnnn nn nnn nn nnnnnnnnnn. Nnn nnnnnnnnnnnn nnn\, nnnnnnnnnnnnn nn nn
 nnnnn nn nnnn nnnnnnnnnnnn nn nnn nnnnnnnnnnn nn nn nn nnnnnnnnnn nnn nnn 
 nn nnnnnnnn. Nn nnn nnnn nnnnnnnn nnnn nnnnnnnnnnnn nn nnnnn\, nnnnnn nnnn
 nn nnn nnnnnn nnnnnnnnnnn nn nnnnnn n-nnnn\, nnn nnnnnnnnnnn nnnnnn nn nnn
 nnnn nnnn n-nnnn\, nnn nnnnnnnnnnn\, nnn nnn nnnnnn (nnnnnnn nn nnnnn). Nn
 nnnn nnnnnnnnn nnnnnn nn nnnn n-nnnn\, nnnnnnn nn nnnn nnnnnnn nnnnnn nn n
 nnnnnnnn nn n nnnnnnn nn nnnnnnnnnn nnnnnnnnn. Nnn nnnnnnnnnn nnnnnnnnn nn
 nnnnnnnnn nnn nnnnnnnnnnn nnnnnnnnn NNN’n nnnnnnnn nnn nnnnnnnn\, nnnnnn
  nnnnn nn nnn nnnnnnnnn nnnn:\n>>>>>\n>>>>> nnnn://nnn.nnn.nnn/nnnnn/nnnnn
 n-nnnnnnnnnnn\n>>>> Nnnn n-nnnn nnn nnn nnnnnnnnnnn nnn nnnnnnnn nnnn nnn 
 nnn nnnnnnnnnn nn nnnnnn nn nnnn nn nn nnnnnnnnn nnn nnn nnnnnnn nnnnnnnnn
 nn nnnn nn nnnnnnnnnnnn\, nnnnnnnnnn\, nnnnnn nnnnnnnnnnn\, nn nnnnnnn nn 
 nnnnn nnnnnnnnnnnn nn nnn nn nnnnnnnnnn. Nnn nnnnnnnnnnnn nnn\, nnnnnnnnnn
 nnn nn nnnnnnn nn nnnn nnnnnnnnnnnn nn nnn nnnnnnnnnnn nn nn nn nnnnnnnnnn
  nnn nnn nn nnnnnnnn. Nn nnn nnnn nnnnnnnn nnnn nnnnnnnnnnnn nn nnnnn\, nn
 nnnn nnnnnn nnn nnnnnn nnnnnnnnnnn nn nnnnnn n-nnnn\, nnn nnnnnnnnnnn nnnn
 nn nn nnnnnnn nnnn n-nnnn\, nnn nnnnnnnnnnn\, nnn nnn nnnnnn (nnnnnnn nn n
 nnnn). Nnnnnn nnnnnnnnn nnnnnn nn nnnn n-nnnn\, nnnnnnn nn nnnn nnnnnnn nn
 nnnn nn nnnnnnnnn nn n nnnnnnn nn nnnnnnnnnn nnnnnnnnn. Nnn nnnnnnnnnn nnn
 nnnnnn nnnnnnnnnnn nnn nnnnnnnnnnn nnnnnnnnn NNN’n nnnnnnnn nnn nnnnnnnn
 \, nnnnnn nnnnn nn nnn nnnnnnnnn nnnn:\n>>>>\n>>>> nnnn://nnn.nnn.nnn/nnnn
 n/nnnnnn-nnnnnnnnnnn\n>>> Nnnn n-nnnn nnn nnn nnnnnnnnnnn nnn nnnnnnnn nnn
 n nnn nnn nnnnnnnnnn nn nnnnnn nn nnnn nn nn nnnnnnnnn nnn nnn nnnnnnn nnn
 nnnnnnnn nnnn nn nnnnnnnnnnnn\, nnnnnnnnnn\, nnnnnn nnnnnnnnnnn\, nn nnnnn
 nn nn nnnnn nnnnnnnnnnnn nn nnn nn nnnnnnnnnn. Nnn nnnnnnnnnnnn nnn\, nnnn
 nnnnnnnnn nn nnnnnnn nn nnnn nnnnnnnnnnnn nn nnn nnnnnnnnnnn nn nn nn nnnn
 nnnnnn nnn nnn nn nnnnnnnn. Nn nnn nnnn nnnnnnnn nnnn nnnnnnnnnnnn nn nnnn
 n\, nnnnnn nnnnnn nnn nnnnnn nnnnnnnnnnn nn nnnnnn n-nnnn\, nnn nnnnnnnnnn
 n nnnnnn nn nnnnnnn nnnn n-nnnn\, nnn nnnnnnnnnnn\, nnn nnn nnnnnn (nnnnnn
 n nn nnnnn). Nnnnnn nnnnnnnnn nnnnnn nn nnnn n-nnnn\, nnnnnnn nn nnnn nnnn
 nnn nnnnnn nn nnnnnnnnn nn n nnnnnnn nn nnnnnnnnnn nnnnnnnnn. Nnn nnnnnnnn
 nn nnnnnnnnn nnnnnnnnnnn nnn nnnnnnnnnnn nnnnnnnnn NNN’n nnnnnnnn nnn nn
 nnnnnn\, nnnnnn nnnnn nn nnn nnnnnnnnn nnnn:\n>>>\n>>> nnnn://nnn.nnn.nnn/
 nnnnn/nnnnnn-nnnnnnnnnnn\n>> Nnnn n-nnnn nnn nnn nnnnnnnnnnn nnn nnnnnnnn 
 nnnn nnn nnn nnnnnnnnnn nn nnnnnn nn nnnn nn nn nnnnnnnnn nnn nnn nnnnnnn 
 nnnnnnnnnnn nnnn nn nnnnnnnnnnnn\, nnnnnnnnnn\, nnnnnn nnnnnnnnnnn\, nn nn
 nnnnn nn nnnnn nnnnnnnnnnnn nn nnn nn nnnnnnnnnn. Nnn nnnnnnnnnnnn nnn\, n
 nnnnnnnnnnnn nn nnnnnnn nn nnnn nnnnnnnnnnnn nn nnn nnnnnnnnnnn nn nn nn n
 nnnnnnnnn nnn nnn nn nnnnnnnn. Nn nnn nnnn nnnnnnnn nnnn nnnnnnnnnnnn nn n
 nnnn\, nnnnnn nnnnnn nnn nnnnnn nnnnnnnnnnn nn nnnnnn n-nnnn\, nnn nnnnnnn
 nnnn nnnnnn nn nnnnnnn nnnn n-nnnn\, nnn nnnnnnnnnnn\, nnn nnn nnnnnn (nnn
 nnnn nn nnnnn). Nnnnnn nnnnnnnnn nnnnnn nn nnnn n-nnnn\, nnnnnnn nn nnnn n
 nnnnnn nnnnnn nn nnnnnnnnn nn n nnnnnnn nn nnnnnnnnnn nnnnnnnnn. Nnn nnnnn
 nnnnn nnnnnnnnn nnnnnnnnnnn nnn nnnnnnnnnnn nnnnnnnnn NNN’n nnnnnnnn nnn
  nnnnnnnn\, nnnnnn nnnnn nn nnn nnnnnnnnn nnnn:\n>>\n>> nnnn://nnn.nnn.nnn
 /nnnnn/nnnnnn-nnnnnnnnnnn\n>>\n>>\n> Nnnn n-nnnn nnn nnn nnnnnnnnnnn nnn n
 nnnnnnn nnnn nnn nnn nnnnnnnnnn nn nnnnnn nn nnnn nn nn nnnnnnnnn nnn nnn 
 nnnnnnn nnnnnnnnnnn nnnn nn nnnnnnnnnnnn\, nnnnnnnnnn\, nnnnnn nnnnnnnnnnn
 \, nn nnnnnnn nn nnnnn nnnnnnnnnnnn nn nnn nn nnnnnnnnnn. Nnn nnnnnnnnnnnn
  nnn\, nnnnnnnnnnnnn nn nnnnnnn nn nnnn nnnnnnnnnnnn nn nnn nnnnnnnnnnn nn
  nn nn nnnnnnnnnn nnn nnn nn nnnnnnnn. Nn nnn nnnn nnnnnnnn nnnn nnnnnnnnn
 nnn nn nnnnn\, nnnnnn nnnnnn nnn nnnnnn nnnnnnnnnnn nn nnnnnn n-nnnn\, nnn
  nnnnnnnnnnn nnnnnn nn nnnnnnn nnnn n-nnnn\, nnn nnnnnnnnnnn\, nnn nnn nnn
 nnn (nnnnnnn nn nnnnn). Nnnnnn nnnnnnnnn nnnnnn nn nnnn n-nnnn\, nnnnnnn n
 n nnnn nnnnnnn nnnnnn nn nnnnnnnnn nn n nnnnnnn nn nnnnnnnnnn nnnnnnnnn. N
 nn nnnnnnnnnn nnnnnnnnn nnnnnnnnnnn nnn nnnnnnnnnnn nnnnnnnnn NNN’n nnnn
 nnnn nnn nnnnnnnn\, nnnnnn nnnnn nn nnn nnnnnnnnn nnnn:\n>\n> nnnn://nnn.n
 nn.nnn/nnnnn/nnnnnn-nnnnnnnnnnn\n\nNnnn n-nnnn nnn nnn nnnnnnnnnnn nnn nnn
 nnnnn nnnn nnn nnn nnnnnnnnnn nn nnnnnn nn nnnn nn nn nnnnnnnnn nnn nnn nn
 nnnnn nnnnnnnnnnn nnnn nn nnnnnnnnnnnn\, nnnnnnnnnn\, nnnnnn nnnnnnnnnnn\,
  nn nnnnnnn nn nnnnn nnnnnnnnnnnn nn nnn nn nnnnnnnnnn. Nnn nnnnnnnnnnnn n
 nn\, nnnnnnnnnnnnn nn nnnnnnn nn nnnn nnnnnnnnnnnn nn nnn nnnnnnnnnnn nn n
 n nn nnnnnnnnnn nnn nnn nn nnnnnnnn. Nn nnn nnnn nnnnnnnn nnnn nnnnnnnnnnn
 n nn nnnnn\, nnnnnn nnnnnn nnn nnnnnn nnnnnnnnnnn nn nnnnnn n-nnnn\, nnn n
 nnnnnnnnnn nnnnnn nn nnnnnnn nnnn n-nnnn\, nnn nnnnnnnnnnn\, nnn nnn nnnnn
 n (nnnnnnn nn nnnnn). Nnnnnn nnnnnnnnn nnnnnn nn nnnn n-nnnn\, nnnnnnn nn 
 nnnn nnnnnnn nnnnnn nn nnnnnnnnn nn n nnnnnnn nn nnnnnnnnnn nnnnnnnnn. Nnn
  nnnnnnnnnn nnnnnnnnn nnnnnnnnnnn nnn nnnnnnnnnnn nnnnnnnnn NNN’n nnnnnn
 nn nnn nnnnnnnn\, nnnnnn nnnnn nn nnn nnnnnnnnn nnnn:\n\nnnnn://nnn.nnn.nn
 n/nnnnn/nnnnnn-nnnnnnnnnnn\n

bug#24315: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

peder@klingenberg.no (Peder O. Klingenberg) writes:

> I recently started using gnus-icalendar to handle meeting invitations.
> Someone sent me an invitation that included a rather long email thread
> in typical outlook style, and gnus errored out when trying to display
> the article.
>
> The backtrace showed that the error comes from the call
>   (re-search-forward  "\\(.*\\)\\(\r?\n[ \t].*\\)*" nil t)
> in icalendar--read-element (nil nil).

This looks like it would be an inefficient regexp, since it has nested
stars.  I suspect searching for "^[ \t]" multiple times would be more
efficient than trying to match multiple lines in a single regexp.

>
> Strangely, the same recipe does not reproduce the error on os x, with a
> build from master from 2016-08-12, but on my GNU/Linux machine, which is
> where I run gnus and need this to work, it is 100% reproducible.  I am
> on Kubuntu 14.04.5, emacs is compiled with gcc (Ubuntu
> 4.8.4-2ubuntu1~14.04.3) 4.8.4.
>
> Is this reproducible for anyone else, or do I need to dig deeper into my
> environment?  Any ideas on where to start digging?

(I'm also on GNU/Linux, Arch) I get the same max-specpdl-size error with
25.1.50, with 24.5 (and below) I get (error "Stack overflow in regexp
matcher") and with 25.1-rc1 I get an assertion failure:

character.h:703: Emacs fatal error: assertion failed: CHAR_VALID_P (ch)

Thread 1 "emacs" hit Breakpoint 1, terminate_due_to_signal (sig=6, backtrace_limit=2147483647)
    at emacs.c:354
354	  signal (sig, SIG_DFL);
(gdb) bt
#0  terminate_due_to_signal (sig=6, backtrace_limit=2147483647) at emacs.c:354
#1  0x00000000005fa92d in die (msg=0x722518 "CHAR_VALID_P (ch)", file=0x72250c "character.h", 
    line=703) at alloc.c:7223
#2  0x000000000056b841 in char_table_translate (obj=20656133, ch=4194341) at character.h:703
#3  0x00000000005e9d33 in re_match_2_internal (bufp=0xd68260 <searchbufs+32>, 
    string1=0x1a9a970 " \202\245\001", size1=0, string2=0x1a9a970 " \202\245\001", size2=40918, 
    pos=0, regs=0xd69e70 <search_regs>, stop=40918) at regex.c:5430
#4  0x00000000005e782c in re_search_2 (bufp=0xd68260 <searchbufs+32>, str1=0x1a9a970 " \202\245\001", 
    size1=0, str2=0x1a9a970 " \202\245\001", size2=40918, startpos=0, range=40918, 
    regs=0xd69e70 <search_regs>, stop=40918) at regex.c:4446
#5  0x00000000005d6447 in search_buffer (string=14974596, pos=1, pos_byte=1, lim=40891, 
    lim_byte=40919, n=1, RE=1, trt=20656133, inverse_trt=20532565, posix=false) at search.c:1265
#6  0x00000000005d5be2 in search_command (string=14974596, bound=0, noerror=44448, count=0, 
    direction=1, RE=1, posix=false) at search.c:1058
#7  0x00000000005d99a7 in Fre_search_forward (regexp=14974596, bound=0, noerror=44448, count=0)
    at search.c:2264
#8  0x000000000061cac3 in eval_sub (form=26022259) at eval.c:2176
#9  0x000000000061c124 in Feval (form=26022259, lexical=0) at eval.c:1988
#10 0x000000000061e052 in Ffuncall (nargs=3, args=0x7fffffffd678) at eval.c:2696
#11 0x0000000000668285 in exec_byte_code (bytestr=10946844, vector=10946877, maxdepth=30, 
    args_template=2054, nargs=2, args=0x7fffffffdc90) at bytecode.c:880
#12 0x000000000061e958 in funcall_lambda (fun=10946789, nargs=2, arg_vector=0x7fffffffdc80)
    at eval.c:2855
#13 0x000000000061e2a1 in Ffuncall (nargs=3, args=0x7fffffffdc78) at eval.c:2742
#14 0x000000000061488d in Ffuncall_interactively (nargs=3, args=0x7fffffffdc78) at callint.c:252
#15 0x000000000061dee7 in Ffuncall (nargs=4, args=0x7fffffffdc70) at eval.c:2673
#16 0x000000000061d338 in Fapply (nargs=3, args=0x7fffffffdd50) at eval.c:2321
#17 0x0000000000614d93 in Fcall_interactively (function=4198576, record_flag=0, keys=14640533)
    at callint.c:389
#18 0x000000000061e08d in Ffuncall (nargs=4, args=0x7fffffffdfd8) at eval.c:2700
#19 0x0000000000668285 in exec_byte_code (bytestr=10951116, vector=10951149, maxdepth=54, 
    args_template=4102, nargs=1, args=0x7fffffffe530) at bytecode.c:880
#20 0x000000000061e958 in funcall_lambda (fun=10951069, nargs=1, arg_vector=0x7fffffffe528)
    at eval.c:2855
#21 0x000000000061e2a1 in Ffuncall (nargs=2, args=0x7fffffffe520) at eval.c:2742
#22 0x000000000061d9e5 in call1 (fn=14784, arg1=4198576) at eval.c:2552
#23 0x00000000005734ce in command_loop_1 () at keyboard.c:1479
#24 0x000000000061a7c1 in internal_condition_case (bfun=0x572cb4 <command_loop_1>, handlers=19056, 
    hfun=0x572346 <cmd_error>) at eval.c:1309
#25 0x00000000005728f6 in command_loop_2 (ignore=0) at keyboard.c:1107
#26 0x0000000000619d90 in internal_catch (tag=45840, func=0x5728cd <command_loop_2>, arg=0)
    at eval.c:1074
#27 0x0000000000572898 in command_loop () at keyboard.c:1086
#28 0x0000000000571e36 in recursive_edit_1 () at keyboard.c:692
#29 0x0000000000572036 in Frecursive_edit () at keyboard.c:763
#30 0x000000000056fde3 in main (argc=3, argv=0x7fffffffe9d8) at emacs.c:1626

Lisp Backtrace:
"re-search-forward" (0xffffd470)
"eval" (0xffffd680)
"eval-expression" (0xffffdc80)
"funcall-interactively" (0xffffdc78)
"call-interactively" (0xffffdfe0)
"command-execute" (0xffffe528)


>
>
>
> In GNU Emacs 25.1.50.2 (x86_64-unknown-linux-gnu, X toolkit, Xaw3d scroll bars)
>  of 2016-08-11 built on luna
> Repository revision: 36a57c55f2b855bca704c26bf7d787b7b471fe16
> Windowing system distributor 'The X.Org Foundation', version 11.0.11604000
> System Description:	Ubuntu 14.04.5 LTS
>
> Recent messages:
> scroll-down-command: Beginning of buffer [3 times]
> (Shell command succeeded with no output) [91 times]
> Back to top level
> When done with this frame, type C-x 5 0
> Undo! [3 times]
> calendar-event-problem.txt changed on disk; really edit the buffer? (y, n, r or C-h) r
> ask-user-about-supersession-threat: File reverted: /home/pok/tmp/calendar-event-problem.txt
> previous-line: Beginning of buffer [3 times]
> Entering debugger...
> Back to top level
>
> Configured using:
>  'configure --prefix=/usr/local/emacs-git
>  --enable-locallisppath=/etc/emacs24:/etc/emacs:/usr/local/share/emacs/site-lisp/:/usr/share/emacs/site-lisp
>  --with-x=yes --with-x-toolkit=lucid --with-toolkit-scroll-bars
>  --with-pop=yes'
>
> Configured features:
> XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GCONF
> GSETTINGS NOTIFY LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT
> ZLIB TOOLKIT_SCROLL_BARS LUCID X11
>
> Important settings:
>   value of $LC_MONETARY: en_US.UTF-8
>   value of $LC_NUMERIC: en_US.UTF-8
>   value of $LANG: en_US.UTF-8
>   locale-coding-system: utf-8-unix
>
> Major mode: Text
>
> Minor modes in effect:
>   csv-field-index-mode: t
>   magit-auto-revert-mode: t
>   global-git-commit-mode: t
>   async-bytecomp-package-mode: t
>   shell-dirtrack-mode: t
>   diff-auto-refine-mode: t
>   show-paren-mode: t
>   yas-global-mode: t
>   yas-minor-mode: t
>   ido-everywhere: t
>   global-eldoc-mode: t
>   electric-indent-mode: t
>   mouse-wheel-mode: t
>   menu-bar-mode: t
>   file-name-shadow-mode: t
>   global-font-lock-mode: t
>   font-lock-mode: t
>   auto-composition-mode: t
>   auto-encryption-mode: t
>   auto-compression-mode: t
>   line-number-mode: t
>   auto-fill-function: do-auto-fill
>   transient-mark-mode: t
>
> Load-path shadows:
> /home/pok/.emacs.d/elpa/org-20160822/org-macs hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-macs
> /home/pok/.emacs.d/elpa/org-20160822/ob-dot hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-dot
> /home/pok/.emacs.d/elpa/org-20160822/ob-mscgen hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-mscgen
> /home/pok/.emacs.d/elpa/org-20160822/ob-clojure hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-clojure
> /home/pok/.emacs.d/elpa/org-20160822/org-install hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-install
> /home/pok/.emacs.d/elpa/org-20160822/org-faces hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-faces
> /home/pok/.emacs.d/elpa/org-20160822/ob-scheme hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-scheme
> /home/pok/.emacs.d/elpa/org-20160822/ob-haskell hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-haskell
> /home/pok/.emacs.d/elpa/org-20160822/ob-sqlite hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-sqlite
> /home/pok/.emacs.d/elpa/org-20160822/ox-texinfo hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-texinfo
> /home/pok/.emacs.d/elpa/org-20160822/ob-perl hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-perl
> /home/pok/.emacs.d/elpa/org-20160822/ob-makefile hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-makefile
> /home/pok/.emacs.d/elpa/org-20160822/ob-io hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-io
> /home/pok/.emacs.d/elpa/org-20160822/org-docview hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-docview
> /home/pok/.emacs.d/elpa/org-20160822/org-irc hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-irc
> /home/pok/.emacs.d/elpa/org-20160822/ob-C hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-C
> /home/pok/.emacs.d/elpa/org-20160822/ox-man hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-man
> /home/pok/.emacs.d/elpa/org-20160822/org-capture hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-capture
> /home/pok/.emacs.d/elpa/org-20160822/ob-table hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-table
> /home/pok/.emacs.d/elpa/org-20160822/ob-sql hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-sql
> /home/pok/.emacs.d/elpa/org-20160822/org-list hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-list
> /home/pok/.emacs.d/elpa/org-20160822/org-crypt hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-crypt
> /home/pok/.emacs.d/elpa/org-20160822/org-attach hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-attach
> /home/pok/.emacs.d/elpa/org-20160822/org-colview hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-colview
> /home/pok/.emacs.d/elpa/org-20160822/org-id hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-id
> /home/pok/.emacs.d/elpa/org-20160822/ob-plantuml hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-plantuml
> /home/pok/.emacs.d/elpa/org-20160822/ox-publish hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-publish
> /home/pok/.emacs.d/elpa/org-20160822/ob-tangle hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-tangle
> /home/pok/.emacs.d/elpa/org-20160822/org hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org
> /home/pok/.emacs.d/elpa/org-20160822/org-indent hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-indent
> /home/pok/.emacs.d/elpa/org-20160822/ox-ascii hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-ascii
> /home/pok/.emacs.d/elpa/org-20160822/ob-matlab hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-matlab
> /home/pok/.emacs.d/elpa/org-20160822/org-gnus hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-gnus
> /home/pok/.emacs.d/elpa/org-20160822/org-feed hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-feed
> /home/pok/.emacs.d/elpa/org-20160822/org-clock hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-clock
> /home/pok/.emacs.d/elpa/org-20160822/org-element hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-element
> /home/pok/.emacs.d/elpa/org-20160822/ob-screen hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-screen
> /home/pok/.emacs.d/elpa/org-20160822/ob hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob
> /home/pok/.emacs.d/elpa/org-20160822/org-mobile hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-mobile
> /home/pok/.emacs.d/elpa/org-20160822/ox-latex hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-latex
> /home/pok/.emacs.d/elpa/org-20160822/ob-lisp hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-lisp
> /home/pok/.emacs.d/elpa/org-20160822/ob-calc hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-calc
> /home/pok/.emacs.d/elpa/org-20160822/org-bibtex hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-bibtex
> /home/pok/.emacs.d/elpa/org-20160822/org-macro hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-macro
> /home/pok/.emacs.d/elpa/org-20160822/org-agenda hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-agenda
> /home/pok/.emacs.d/elpa/org-20160822/org-entities hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-entities
> /home/pok/.emacs.d/elpa/org-20160822/ob-css hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-css
> /home/pok/.emacs.d/elpa/org-20160822/ob-R hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-R
> /home/pok/.emacs.d/elpa/org-20160822/org-mouse hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-mouse
> /home/pok/.emacs.d/elpa/org-20160822/org-timer hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-timer
> /home/pok/.emacs.d/elpa/org-20160822/ob-java hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-java
> /home/pok/.emacs.d/elpa/org-20160822/ob-gnuplot hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-gnuplot
> /home/pok/.emacs.d/elpa/org-20160822/ob-keys hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-keys
> /home/pok/.emacs.d/elpa/org-20160822/ob-python hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-python
> /home/pok/.emacs.d/elpa/org-20160822/org-compat hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-compat
> /home/pok/.emacs.d/elpa/org-20160822/ob-fortran hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-fortran
> /home/pok/.emacs.d/elpa/org-20160822/ob-exp hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-exp
> /home/pok/.emacs.d/elpa/org-20160822/ox-html hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-html
> /home/pok/.emacs.d/elpa/org-20160822/ob-comint hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-comint
> /home/pok/.emacs.d/elpa/org-20160822/org-src hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-src
> /home/pok/.emacs.d/elpa/org-20160822/ob-scala hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-scala
> /home/pok/.emacs.d/elpa/org-20160822/org-version hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-version
> /home/pok/.emacs.d/elpa/org-20160822/org-loaddefs hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-loaddefs
> /home/pok/.emacs.d/elpa/org-20160822/ob-emacs-lisp hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-emacs-lisp
> /home/pok/.emacs.d/elpa/org-20160822/org-datetree hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-datetree
> /home/pok/.emacs.d/elpa/org-20160822/ob-asymptote hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-asymptote
> /home/pok/.emacs.d/elpa/org-20160822/ox-org hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-org
> /home/pok/.emacs.d/elpa/org-20160822/org-habit hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-habit
> /home/pok/.emacs.d/elpa/org-20160822/ox hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox
> /home/pok/.emacs.d/elpa/org-20160822/org-w3m hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-w3m
> /home/pok/.emacs.d/elpa/org-20160822/ob-lob hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-lob
> /home/pok/.emacs.d/elpa/org-20160822/org-protocol hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-protocol
> /home/pok/.emacs.d/elpa/org-20160822/ob-shen hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-shen
> /home/pok/.emacs.d/elpa/org-20160822/ob-maxima hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-maxima
> /home/pok/.emacs.d/elpa/org-20160822/ob-ref hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-ref
> /home/pok/.emacs.d/elpa/org-20160822/ob-org hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-org
> /home/pok/.emacs.d/elpa/org-20160822/ob-latex hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-latex
> /home/pok/.emacs.d/elpa/org-20160822/ox-md hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-md
> /home/pok/.emacs.d/elpa/org-20160822/org-mhe hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-mhe
> /home/pok/.emacs.d/elpa/org-20160822/ob-ledger hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-ledger
> /home/pok/.emacs.d/elpa/org-20160822/org-archive hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-archive
> /home/pok/.emacs.d/elpa/org-20160822/org-ctags hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-ctags
> /home/pok/.emacs.d/elpa/org-20160822/org-info hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-info
> /home/pok/.emacs.d/elpa/org-20160822/ob-octave hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-octave
> /home/pok/.emacs.d/elpa/org-20160822/org-pcomplete hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-pcomplete
> /home/pok/.emacs.d/elpa/org-20160822/ob-ditaa hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-ditaa
> /home/pok/.emacs.d/elpa/org-20160822/org-plot hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-plot
> /home/pok/.emacs.d/elpa/org-20160822/ob-eval hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-eval
> /home/pok/.emacs.d/elpa/org-20160822/org-table hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-table
> /home/pok/.emacs.d/elpa/org-20160822/ox-beamer hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-beamer
> /home/pok/.emacs.d/elpa/org-20160822/ox-odt hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-odt
> /home/pok/.emacs.d/elpa/org-20160822/ob-core hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-core
> /home/pok/.emacs.d/elpa/org-20160822/org-inlinetask hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-inlinetask
> /home/pok/.emacs.d/elpa/org-20160822/org-rmail hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-rmail
> /home/pok/.emacs.d/elpa/org-20160822/ox-icalendar hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ox-icalendar
> /home/pok/.emacs.d/elpa/org-20160822/ob-lilypond hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-lilypond
> /home/pok/.emacs.d/elpa/org-20160822/ob-picolisp hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-picolisp
> /home/pok/.emacs.d/elpa/org-20160822/ob-sass hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-sass
> /home/pok/.emacs.d/elpa/org-20160822/org-eshell hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-eshell
> /home/pok/.emacs.d/elpa/org-20160822/ob-awk hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-awk
> /home/pok/.emacs.d/elpa/org-20160822/ob-ruby hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-ruby
> /home/pok/.emacs.d/elpa/org-20160822/ob-ocaml hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-ocaml
> /home/pok/.emacs.d/elpa/org-20160822/ob-js hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/ob-js
> /home/pok/.emacs.d/elpa/org-20160822/org-footnote hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-footnote
> /home/pok/.emacs.d/elpa/org-20160822/org-bbdb hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/org/org-bbdb
> /usr/share/emacs24/site-lisp/latex-cjk-thai/thai-word hides /usr/local/emacs-git/share/emacs/25.1.50/lisp/language/thai-word
>
> Features:
> (shadow emacsbug gnus-dup gnus-draft nndoc cus-edit ess-omg-l
> ess-toolbar ess-mouse mouseme ess-menu ess-swv ess-noweb essd-els
> ess-sas-d ess-sas-l ess-sas-a ess-sta-d ess-sta-l make-regexp ess-sp6-d
> ess-dde ess-sp3-d ess-julia julia-mode ess-r-d ess-r-syntax
> ess-r-completion ess-roxy essddr hideshow ess-help reporter
> ess-r-package ess-s-l ess-site ess ess-inf ess-tracebug ess-mode
> ess-noweb-mode ess-custom executable ess-generics ess-utils ess-bugs-l
> ess-compat ess-lsp-l map esh-var esh-io esh-cmd esh-opt esh-ext esh-proc
> esh-arg esh-groups eshell esh-module esh-mode esh-util dired-x autoload
> tar-mode ffap php-mode flymake cc-mode cc-fonts cc-guess cc-menus
> cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs hippie-exp vc
> vc-dispatcher url-queue grep pulse vc-cvs slime-indentation
> slime-cl-indent cl-indent slime-fancy slime-trace-dialog
> slime-fontifying-fu slime-package-fu slime-references
> slime-compiler-notes-tree slime-scratch slime-presentations bridge
> slime-macrostep macrostep slime-mdot-fu slime-enclosing-context
> slime-fuzzy slime-fancy-trace slime-fancy-inspector slime-c-p-c
> slime-editing-commands slime-autodoc slime-repl elp slime-parse slime
> lisp-mnt gud apropos etags xref project arc-mode archive-mode hyperspec
> csv-mode magit-blame magit-stash magit-bisect magit-remote magit-commit
> magit-sequence magit magit-apply magit-wip magit-log magit-diff
> smerge-mode magit-core magit-autorevert autorevert filenotify
> magit-process magit-popup magit-mode magit-git magit-section magit-utils
> git-commit log-edit pcvs-util with-editor async-bytecomp async tramp-sh
> tramp tramp-compat tramp-loaddefs trampver ucs-normalize shell dash
> eieio-opt speedbar sb-image ezimage dframe cal-china lunar solar cal-dst
> cal-bahai cal-islam cal-hebrew holidays hol-loaddefs cal-iso bookmark pp
> tabify org-rmail org-mhe org-irc org-info org-gnus org-docview doc-view
> jka-compr image-mode org-bbdb org-w3m org-table org-archive org-clock
> org-indent ob-ditaa ob-plantuml org-bibtex org-element avl-tree bibtex
> org-colview org-crypt org-habit org-agenda help-fns radix-tree misearch
> multi-isearch thingatpt debbugs-gnu add-log debbugs soap-client warnings
> rng-xsd rng-dt rng-util xsd-regexp debug cus-start cus-load mailalias
> bbdb-message footnote ecomplete flow-fill vc-git diff-mode sort
> gnus-cite shr-color color shr svg dom browse-url mail-extr gnus-async
> gnus-bcklg qp gnus-ml disp-table gnus-topic mm-archive url-http url-gw
> url-cache url-auth pop3 nnrss xml mm-url url url-proxy url-privacy
> url-expand url-methods url-history url-cookie url-domsuf url-util
> nndraft nnmh nnml utf-7 bbdb-gnus epa-file network-stream nsm starttls
> gnus-agent gnus-srvr gnus-score score-mode nnvirtual gnus-msg nntp
> gnus-cache pok-gnus gnus-icalendar org-capture gnus-art mm-uu mml2015
> mm-view mml-smime smime dig mailcap icalendar diary-lib diary-loaddefs
> nnir gnus-sum gnus-group gnus-undo bbdb-mua bbdb-com crm bbdb bbdb-site
> timezone gnus-start gnus-cloud nnimap nnmail mail-source tls gnutls utf7
> netrc nnoo parse-time gnus-spec gnus-int gnus-range message sendmail
> puny dired dired-loaddefs rfc822 mml mml-sec epa derived epg mm-decode
> mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
> gnus-win gnus nnheader subr-x gnus-util rmail rmail-loaddefs rfc2047
> rfc2045 ietf-drums mail-utils wid-edit linum paren paredit pok-init
> bugz-mode yasnippet org advice org-macro org-footnote org-pcomplete
> pcomplete org-list org-faces org-entities time-date noutline outline
> org-version ob-emacs-lisp ob ob-tangle ob-ref ob-lob ob-table ob-exp
> org-src ob-keys ob-comint ob-core ob-eval org-compat org-macs
> org-loaddefs format-spec find-func cal-menu calendar cal-loaddefs gedcom
> slime-autoloads reftex reftex-loaddefs reftex-vars edmacro kmacro ido
> server filladapt dmacro mm-util mail-prsvr cl compile comint ansi-color
> ring finder-inf tex-site info package epg-config url-handlers url-parse
> auth-source cl-seq eieio eieio-core cl-macs eieio-loaddefs
> password-cache url-vars seq byte-opt gv bytecomp byte-compile cl-extra
> help-mode easymenu cconv cl-loaddefs pcase cl-lib emacs-goodies-el
> emacs-goodies-custom emacs-goodies-loaddefs easy-mmode mule-util tooltip
> eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel
> term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
> regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode
> prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
> mouse jit-lock font-lock syntax facemenu font-core term/tty-colors frame
> cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai
> tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
> slovak czech european ethiopic indian cyrillic chinese charscript
> case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
> cl-preloaded nadvice loaddefs button faces cus-face macroexp files
> text-properties overlay sha1 md5 base64 format env code-pages mule
> custom widget hashtable-print-readable backquote dbusbind inotify
> dynamic-setting system-font-setting font-render-setting x-toolkit x
> multi-tty make-network-process emacs)
>
> Memory information:
> ((conses 16 2485829 182861)
>  (symbols 48 72236 91)
>  (miscs 40 3269 4672)
>  (strings 32 298119 63000)
>  (string-bytes 1 12887713)
>  (vectors 16 103720)
>  (vector-slots 8 2448384 49706)
>  (floats 8 4985 2034)
>  (intervals 56 225770 627)
>  (buffers 976 190)
>  (heap 1024 299121 -1842968))

bug#24315: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Peder O. Klingenberg]

[-- Attachment #1: Type: text/plain, Size: 1409 bytes --]

On Fri, Aug 26 2016 at 23:35, npostavs@users.sourceforge.net wrote:

> This looks like it would be an inefficient regexp, since it has nested
> stars.  I suspect searching for "^[ \t]" multiple times would be more
> efficient than trying to match multiple lines in a single regexp.

Tried that.  It sort of worked, in that it worked when I directly called
`icalendar--read-event' in a buffer containing the event as manually
extracted from the gnus message.

However, parsing the problematic message in gnus still failed.  It turns
out `gnus-icalendar-event-from-buffer' calls
`icalendar--get-unfolded-buffer' before `icalendar--read-event'.

`icalendar--get-unfolded-buffer' takes care of all those pesky line
continuations, and returns a buffer with each element occupying exactly
one line.  So essentially `icalendar--read-event' was using a regexp to
extract the rest of the line, and that failed with a line nearly 40k
characters long.

The only other in-tree caller of `icalendar--read-event' (excluding
itself) also calls `icalendar--get-unfolded-buffer' first, and indeed,
`icalendar-read-event' already relies on this to take care of line
continuations in element parameters (as opposed to values).

Attached is a patch that uses buffer-substring to accomplish the same
thing as the offending regexp, and documents the dependency on
`icalendar--get-unfolded-buffer'.  This fixes the problem for me.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: bug24315.patch --]
[-- Type: text/x-diff, Size: 1540 bytes --]

From c3ae84dc48a993bb1906d5c9b6b663184e2a9d9d Mon Sep 17 00:00:00 2001
From: "Peder O. Klingenberg" <peder@klingenberg.no>
Date: Tue, 30 Aug 2016 14:44:16 +0200
Subject: [PATCH] Avoid crash in icalendar--read-element

* lisp/calendar/icalendar.el (icalendar--read-element): Stop crashing
when parsing overly long event descriptions.  (Bug#24315)
---
 lisp/calendar/icalendar.el | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/lisp/calendar/icalendar.el b/lisp/calendar/icalendar.el
index 386c554..19930d5 100644
--- a/lisp/calendar/icalendar.el
+++ b/lisp/calendar/icalendar.el
@@ -361,7 +361,8 @@ icalendar--read-element
 INVALUE gives the current iCalendar element we are reading.
 INPARAMS gives the current parameters.....
 This function calls itself recursively for each nested calendar element
-it finds."
+it finds. The current buffer should be an unfolded buffer as returned
+from `icalendar--get-unfolded-buffer'."
   (let (element children line name params param param-name param-value
                 value
                 (continue t))
@@ -391,8 +392,8 @@ icalendar--read-element
       (unless (looking-at ":")
         (error "Oops"))
       (forward-char 1)
-      (re-search-forward  "\\(.*\\)\\(\r?\n[ \t].*\\)*" nil t)
-      (setq value (icalendar--rris "\r?\n[ \t]" "" (match-string 0)))
+      (setq value (buffer-substring (point) (line-end-position)))
+      (end-of-line)
       (setq line (list name params value))
       (cond ((eq name 'BEGIN)
              (setq children
-- 
2.9.3

bug#24315: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

pok@netfonds.no (Peder O. Klingenberg) writes:

> `icalendar--get-unfolded-buffer' takes care of all those pesky line
> continuations, and returns a buffer with each element occupying exactly
> one line.  So essentially `icalendar--read-event' was using a regexp to
> extract the rest of the line, and that failed with a line nearly 40k
> characters long.
>
> The only other in-tree caller of `icalendar--read-event' (excluding
> itself) also calls `icalendar--get-unfolded-buffer' first, and indeed,
> `icalendar-read-event' already relies on this to take care of line
> continuations in element parameters (as opposed to values).
>
> Attached is a patch that uses buffer-substring to accomplish the same
> thing as the offending regexp, and documents the dependency on
> `icalendar--get-unfolded-buffer'.  This fixes the problem for me.

If it works without the regexp, that's even better.

> -it finds."
> +it finds. The current buffer should be an unfolded buffer as returned
            ^
Sentences should be double spaced.

> +from `icalendar--get-unfolded-buffer'."
>    (let (element children line name params param param-name param-value
>                  value
>                  (continue t))
> @@ -391,8 +392,8 @@ icalendar--read-element
>        (unless (looking-at ":")
>          (error "Oops"))
>        (forward-char 1)
> -      (re-search-forward  "\\(.*\\)\\(\r?\n[ \t].*\\)*" nil t)
> -      (setq value (icalendar--rris "\r?\n[ \t]" "" (match-string 0)))
> +      (setq value (buffer-substring (point) (line-end-position)))
> +      (end-of-line)

Might be better to avoid finding the end of line twice (since apparently
40k lines do happen, I guess it's worth thinking a bit about
optimizing):

(let ((start (prog1 (point) (end-of-line))))
  (setq value (buffer-substring start (point))))

bug#24315: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Peder O. Klingenberg]

[-- Attachment #1: Type: text/plain, Size: 757 bytes --]

On Thu, Sep 01 2016 at 21:58, npostavs@users.sourceforge.net wrote:

>> -it finds."
>> +it finds. The current buffer should be an unfolded buffer as returned
>             ^
> Sentences should be double spaced.

Gah!  And I'm usually chided for using old-fashioned double spaces in
prose. :)

> Might be better to avoid finding the end of line twice (since apparently
> 40k lines do happen, I guess it's worth thinking a bit about
> optimizing):

Even finding the the end of the line twice was faster than the regexp
method, subjectively.  On my problematic 40k line, it did not take
noticable time, whereas the regexp caused a bit of a delay even on
shorter events.  But I agree, let's not make emacs do more work than
necessary.  Updated patch attached.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: bug24315.patch --]
[-- Type: text/x-diff, Size: 1563 bytes --]

From b51d09286492ae9be32b3718407174236db54059 Mon Sep 17 00:00:00 2001
From: "Peder O. Klingenberg" <peder@klingenberg.no>
Date: Tue, 30 Aug 2016 14:44:16 +0200
Subject: [PATCH] Avoid crash in icalendar--read-element

* lisp/calendar/icalendar.el (icalendar--read-element): Stop crashing
when parsing overly long event descriptions.  (Bug#24315)
---
 lisp/calendar/icalendar.el | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/lisp/calendar/icalendar.el b/lisp/calendar/icalendar.el
index 386c554..c88f4ab 100644
--- a/lisp/calendar/icalendar.el
+++ b/lisp/calendar/icalendar.el
@@ -361,7 +361,8 @@ icalendar--read-element
 INVALUE gives the current iCalendar element we are reading.
 INPARAMS gives the current parameters.....
 This function calls itself recursively for each nested calendar element
-it finds."
+it finds.  The current buffer should be an unfolded buffer as returned
+from `icalendar--get-unfolded-buffer'."
   (let (element children line name params param param-name param-value
                 value
                 (continue t))
@@ -391,8 +392,9 @@ icalendar--read-element
       (unless (looking-at ":")
         (error "Oops"))
       (forward-char 1)
-      (re-search-forward  "\\(.*\\)\\(\r?\n[ \t].*\\)*" nil t)
-      (setq value (icalendar--rris "\r?\n[ \t]" "" (match-string 0)))
+      (let ((start (point)))
+        (end-of-line)
+        (setq value (buffer-substring start (point))))
       (setq line (list name params value))
       (cond ((eq name 'BEGIN)
              (setq children
-- 
2.9.3

bug#24315: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

# the new bug will be about the specdl error
clone 24315 -1
# this bug is about the bad regex in icalendar
retitle 24315 icalendar--read-element uses inefficient regexp
quit

peder@klingenberg.no (Peder O. Klingenberg) writes:
>  Updated patch attached.
>
[...]
> Subject: [PATCH] Avoid crash in icalendar--read-element
>
> * lisp/calendar/icalendar.el (icalendar--read-element): Stop crashing
> when parsing overly long event descriptions.  (Bug#24315)

Patch looks good, though I think the description shouldn't say "crash",
since the problem is a regex stack overflow, which is just a normal
Emacs error (I've figured out the reason why it's triggering
max-specpdl-size instead of regex stack overflow in master, and will
address it in a separate bug).

bug#24315: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Peder O. Klingenberg]

[-- Attachment #1: Type: text/plain, Size: 288 bytes --]

On Sat, Sep 03 2016 at 10:21, npostavs@users.sourceforge.net wrote:

> Patch looks good, though I think the description shouldn't say "crash",
> since the problem is a regex stack overflow, which is just a normal
> Emacs error

I agree, that was a poor choice of words.  How about this?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: bug24315.patch --]
[-- Type: text/x-diff, Size: 1591 bytes --]

From bd28797ce52c070e59d26483bdf842fd31651de6 Mon Sep 17 00:00:00 2001
From: "Peder O. Klingenberg" <peder@klingenberg.no>
Date: Tue, 30 Aug 2016 14:44:16 +0200
Subject: [PATCH] Avoid error in icalendar--read-element

* lisp/calendar/icalendar.el (icalendar--read-element): Avoid a regex
stack overflow by not using regex to extract values from calendar
events. (Bug#24315)
---
 lisp/calendar/icalendar.el | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/lisp/calendar/icalendar.el b/lisp/calendar/icalendar.el
index 386c554..c88f4ab 100644
--- a/lisp/calendar/icalendar.el
+++ b/lisp/calendar/icalendar.el
@@ -361,7 +361,8 @@ icalendar--read-element
 INVALUE gives the current iCalendar element we are reading.
 INPARAMS gives the current parameters.....
 This function calls itself recursively for each nested calendar element
-it finds."
+it finds.  The current buffer should be an unfolded buffer as returned
+from `icalendar--get-unfolded-buffer'."
   (let (element children line name params param param-name param-value
                 value
                 (continue t))
@@ -391,8 +392,9 @@ icalendar--read-element
       (unless (looking-at ":")
         (error "Oops"))
       (forward-char 1)
-      (re-search-forward  "\\(.*\\)\\(\r?\n[ \t].*\\)*" nil t)
-      (setq value (icalendar--rris "\r?\n[ \t]" "" (match-string 0)))
+      (let ((start (point)))
+        (end-of-line)
+        (setq value (buffer-substring start (point))))
       (setq line (list name params value))
       (cond ((eq name 'BEGIN)
              (setq children
-- 
2.9.3

bug#24315: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

tags 24315 fixed
close 24315 25.2
quit

peder@klingenberg.no (Peder O. Klingenberg) writes:

> On Sat, Sep 03 2016 at 10:21, npostavs@users.sourceforge.net wrote:
>
> I agree, that was a poor choice of words.  How about this?

Thanks, pushed as 55dde6c1

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

# this cloned bug doesn't have a patch yet
tags 24358 - patch
quit

npostavs@users.sourceforge.net writes:

> (I'm also on GNU/Linux, Arch) I get the same max-specpdl-size error with
> 25.1.50, with 24.5 (and below) I get (error "Stack overflow in regexp
> matcher")

The problem is that re_max_failures is set to 40001 (instead of the
original 40000) in main()[1], which is a problem because of the
GROW_FAIL_STACK uses (re_max_failures * TYPICAL_FAILURE_SIZE) as a cap
on the amount to allocate, but ((fail_stack).size * sizeof
(fail_stack_elt_t)) to calculate current allocation.

Since TYPICAL_FAILURE_SIZE = 20 and sizeof (fail_stack_elt_t) = 8, and
it seems that (fail_stack).size grows in increments of 3, when
(fail_stack).avail is 99999 and (fail_stack).size reaches 100002:

(fail_stack).size * sizeof (fail_stack_elt_t) => 800016
  re_max_failures * TYPICAL_FAILURE_SIZE      => 800020

ENSURE_FAIL_STACK(3) then loops indefinitely reallocating a stack of
size 800020 again and again until the record_xmalloc fails to
grow_specdl() (thus the "max-specpdl-size" error).

----------

So we we might want to fix the re_max_failures setting in main, but it
doesn't quite make sense to me that GROW_FAIL_STACK relies on
re_max_failures being a multiple of (sizeof (fail_stack_elt_t)).  At the
definition of TYPICAL_FAILURE_SIZE we have

/* Estimate the size of data pushed by a typical failure stack entry.
   An estimate is all we need, because all we use this for
   is to choose a limit for how big to make the failure stack.  */
/* BEWARE, the value `20' is hard-coded in emacs.c:main().  */
#define TYPICAL_FAILURE_SIZE 20

Why do we use an "estimate" here?  What's wrong with just using
(re_max_failures * sizeof (fail_stack_elt_t)) as the limit?  Or should
the limit actually be (re_max_failures * TYPICAL_FAILURE_SIZE * sizeof
(fail_stack_elt_t))?

-----------

827	      long lim = rlim.rlim_cur;
(gdb) p rlim
$1 = {
  rlim_cur = 8388608, 
  rlim_max = 18446744073709551615
}
(gdb) next
833	      int ratio = 20 * sizeof (char *);
(gdb) 
834	      ratio += ratio / 3;
(gdb) 
837	      int extra = 200000;
(gdb) p ratio
$2 = 213
[...]
(gdb) display ((newlim - extra) / ratio)
1: ((newlim - extra) / ratio) = 40000
(gdb) next
856		  newlim += pagesize - 1;
1: ((newlim - extra) / ratio) = 40000
(gdb) 
857		  if (0 <= rlim.rlim_max && rlim.rlim_max < newlim)
1: ((newlim - extra) / ratio) = 40019
(gdb) 
859		  newlim -= newlim % pagesize;
1: ((newlim - extra) / ratio) = 40019
(gdb) 
861		  if (pagesize <= newlim - lim)
1: ((newlim - extra) / ratio) = 40001
(gdb) undisplay 1
(gdb) next
863		      rlim.rlim_cur = newlim;
(gdb) 
864		      if (setrlimit (RLIMIT_STACK, &rlim) == 0)
(gdb) 
865			lim = newlim;
(gdb) 
870	      re_max_failures = lim < extra ? 0 : min (lim - extra, SIZE_MAX) / ratio;
(gdb) 
875	  stack_bottom = &stack_bottom_variable;
(gdb) p re_max_failures
$3 = 40001

-----------

[1]: This was the case since 9d356f62 2016-05-27 "Robustify stack-size calculation"

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

npostavs@users.sourceforge.net writes:
>
>> (I'm also on GNU/Linux, Arch) I get the same max-specpdl-size error with
>> 25.1.50, with 24.5 (and below) I get (error "Stack overflow in regexp
>> matcher")

icalendar--read-element has been fixed, but this still reproduces when
doing (re-search-forward ".*\\(\n.*\\)*" nil t) on the text file given
in the OP.  And I'm still seeing an assertion failure due to what looks
like memory corruption on the emacs-25 branch.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Date: Fri, 07 Oct 2016 20:29:36 -0400
> Cc: 24358@debbugs.gnu.org
> 
> npostavs@users.sourceforge.net writes:
> >
> >> (I'm also on GNU/Linux, Arch) I get the same max-specpdl-size error with
> >> 25.1.50, with 24.5 (and below) I get (error "Stack overflow in regexp
> >> matcher")
> 
> icalendar--read-element has been fixed, but this still reproduces when
> doing (re-search-forward ".*\\(\n.*\\)*" nil t) on the text file given
> in the OP.

Isn't that "user error"?

> And I'm still seeing an assertion failure due to what looks like
> memory corruption on the emacs-25 branch.

Details of the assertion?

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

[-- Attachment #1: Type: text/plain, Size: 2720 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: npostavs@users.sourceforge.net
>> Date: Fri, 07 Oct 2016 20:29:36 -0400
>> Cc: 24358@debbugs.gnu.org
>> 
>> npostavs@users.sourceforge.net writes:
>> >
>> >> (I'm also on GNU/Linux, Arch) I get the same max-specpdl-size error with
>> >> 25.1.50, with 24.5 (and below) I get (error "Stack overflow in regexp
>> >> matcher")
>> 
>> icalendar--read-element has been fixed, but this still reproduces when
>> doing (re-search-forward ".*\\(\n.*\\)*" nil t) on the text file given
>> in the OP.
>
> Isn't that "user error"?

Yes, but it should give "Stack overflow in regexp matcher", not overflow
the lisp stack (or assertion failure).

>
>> And I'm still seeing an assertion failure due to what looks like
>> memory corruption on the emacs-25 branch.
>
> Details of the assertion?

(See also https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24358#8)

I tracked the corruption to a malloc call, but I wasn't able to figure
out what's happening there.

I used the following to debug:

Apply the attached bug-24358-hunting.diff and then run

    gdb --args ./emacs -Q -batch -l ~/src/emacs/bug-24358-regex-max-specpdl.el

Where ~/src/emacs/bug-24358-regex-max-specpdl.el is:

    (with-temp-buffer
      (insert-file-contents "~/src/emacs/bug-24358-regex-max-specpdl.txt") ; adjust path
      (goto-char (point-min))
      (re-search-forward ".*\\(\n.*\\)*" nil t))

I show some more excerpts in the attached bug-24358-debug.log, but my
main finding is that string1 of re_match_2_internal is originally:

    string1=0x1835980 "DESCRIPTION;LANGUAGE=

but then it becomes corrupted during a malloc:

Old value = 68 'D'
New value = 0 '\000'
0x00007ffff0cc01a7 in __memset_sse2_unaligned_erms () from /usr/lib/libc.so.6

(gdb) bt 13
#0  0x00007ffff0cc01a7 in __memset_sse2_unaligned_erms () from /usr/lib/libc.so.6
#1  0x00000000006d27f5 in r_alloc_sbrk (size=290816) at ralloc.c:848
#2  0x00000000006ced96 in get_contiguous_space (size=290816, position=0x1833000) at gmalloc.c:476
#3  0x00000000006cf92a in _malloc_internal_nolock (size=163840) at gmalloc.c:844
#4  0x00000000006cfe9d in _malloc_internal (size=163840) at gmalloc.c:927
#5  0x00000000006cff1a in gmalloc (size=163840) at gmalloc.c:951
#6  0x00000000006d14e4 in malloc (size=163840) at gmalloc.c:1827
#7  0x00000000005f3e6b in lmalloc (size=163840) at alloc.c:1414
#8  0x00000000005f3356 in xmalloc (size=163840) at alloc.c:821
#9  0x00000000005f38e4 in record_xmalloc (size=163840) at alloc.c:1038
#10 0x00000000005ee233 in re_match_2_internal (bufp=0xd6d650 <searchbufs+5072>, string1=0x1835980 "", 
    size1=0, string2=0x1835980 "", size2=40918, pos=0, regs=0xd6deb0 <search_regs>, stop=40918)
    at regex.c:5844



[-- Attachment #2: changes to hunt down bug 24358 --]
[-- Type: text/plain, Size: 3199 bytes --]

diff --git i/src/.gdbinit w/src/.gdbinit
index a4e9f70..d17d1ba 100644
--- i/src/.gdbinit
+++ w/src/.gdbinit
@@ -1280,3 +1280,22 @@ commands
   end
   continue
 end
+
+# bug 24315
+break re_match_2_internal if (size2>2000 && size2==stop)
+commands
+  p debug = 1
+  continue
+end
+break debug_spot
+commands
+  watch -l string1[0]
+  disable 4
+  # cond 4 (string1[0] != 'D')
+  # continue
+end
+# break debug_malloc if ((mem <= 0x1834980) && (0x1834980 < mem + size))
+
+
+
+
diff --git i/src/gmalloc.c w/src/gmalloc.c
index 00b8364..5084609 100644
--- i/src/gmalloc.c
+++ w/src/gmalloc.c
@@ -914,6 +914,10 @@ _malloc_internal_nolock (size_t size)
   return result;
 }
 
+void debug_malloc (void* mem, size_t size)
+{
+}
+
 void *
 _malloc_internal (size_t size)
 {
@@ -923,6 +927,7 @@ _malloc_internal (size_t size)
   result = _malloc_internal_nolock (size);
   UNLOCK ();
 
+  debug_malloc (result, size);
   return result;
 }
 
diff --git i/src/regex.c w/src/regex.c
index 164eb46..861b800 100644
--- i/src/regex.c
+++ w/src/regex.c
@@ -828,6 +828,7 @@ extract_number_and_incr (re_char **source)
    interactively.  And if linked with the main program in `main.c' and
    the other test files, you can run the already-written tests.  */
 
+#define DEBUG
 #ifdef DEBUG
 
 /* We use standard I/O for debugging.  */
@@ -838,6 +839,13 @@ extract_number_and_incr (re_char **source)
 
 static int debug = -100000;
 
+static void debug_spot (int fail_stack_avail, const char*string1, const char*string2)
+{
+  extern void r_alloc_check (void);
+  //r_alloc_check ();
+  fail_stack_avail++;
+}
+
 # define DEBUG_STATEMENT(e) e
 # define DEBUG_PRINT(...) if (debug > 0) printf (__VA_ARGS__)
 # define DEBUG_COMPILES_ARGUMENTS
@@ -1172,16 +1180,31 @@ print_double_string (re_char *where, re_char *string1, ssize_t size1,
     printf ("(null)");
   else
     {
+      int i;
       if (FIRST_STRING_P (where))
 	{
-	  for (this_char = where - string1; this_char < size1; this_char++)
-	    putchar (string1[this_char]);
+	  for (i = 0, this_char = where - string1; this_char < size1; i++, this_char++)
+            {
+              if (i > 20)
+                {
+                  putchar ('.'); putchar ('.'); putchar ('.');
+                  break;
+                }
+              putchar (string1[this_char]);
+            }
 
 	  where = string2;
 	}
 
-      for (this_char = where - string2; this_char < size2; this_char++)
-	putchar (string2[this_char]);
+      for (i = 0, this_char = where - string2; this_char < size2; i++, this_char++)
+        {
+          if (i > 20)
+            {
+              putchar ('.'); putchar ('.'); putchar ('.');
+              break;
+            }
+          putchar (string2[this_char]);
+        }
     }
 }
 
@@ -1533,6 +1556,7 @@ while (REMAINING_AVAIL_SLOTS <= space) {				\
      of 0 + -1 isn't done as unsigned.  */				\
   									\
   DEBUG_STATEMENT (nfailure_points_pushed++);				\
+  if (debug > 0) debug_spot((fail_stack).avail, string1,string2);              \
   DEBUG_PRINT ("\nPUSH_FAILURE_POINT:\n");				\
   DEBUG_PRINT ("  Before push, next avail: %zd\n", (fail_stack).avail);	\
   DEBUG_PRINT ("			size: %zd\n", (fail_stack).size);\

[-- Attachment #3: gdb session excerpts --]
[-- Type: text/plain, Size: 7321 bytes --]

The compiled pattern is: The string to match is: "DESCRIPTION;LANGUAGE=..."

0x144aa80: EXECUTING on_failure_jump_smart 4 (to 0x144aa87).
  smart default => slow loop.

0x144aa80: EXECUTING on_failure_jump 4 (to 0x144aa87):

Thread 1 "emacs" hit Breakpoint 4, debug_spot (fail_stack_avail=0, 
    string1=0x1835980 "DESCRIPTION;LANGUAGE=en-US:Nn Nnnnn\\,\\n\\nNnnnnnnnn nnn nnn nnnnnn nn nnnnnn\n nnnnnnn nnnn nnnnnnnnn. N nnnn nnnnnnnnn nn nnn nnnnnnnn nnnnnnn nnn nn nn\n nn nn-nnnnnnn nn Nnnnnnn nn 99.99 NNNN\\n\\nNnnn "..., 
    string2=0x1835980 "DESCRIPTION;LANGUAGE=en-US:Nn Nnnnn\\,\\n\\nNnnnnnnnn nnn nnn nnnnnn nn nnnnnn\n nnnnnnn nnnn nnnnnnnnn. N nnnn nnnnnnnnn nn nnn nnnnnnnn nnnnnnn nnn nn nn\n nn nn-nnnnnnn nn Nnnnnnn nn 99.99 NNNN\\n\\nNnnn "...) at regex.c:846
846	  fail_stack_avail++;
Hardware watchpoint 5: -location string1[0]
(gdb) bt 5
#0  debug_spot (fail_stack_avail=0, 
    string1=0x1835980 "DESCRIPTION;LANGUAGE=en-US:Nn Nnnnn\\,\\n\\nNnnnnnnnn nnn nnn nnnnnn nn nnnnnn\n nnnnnnn nnnn nnnnnnnnn. N nnnn nnnnnnnnn nn nnn nnnnnnnn nnnnnnn nnn nn nn\n nn nn-nnnnnnn nn Nnnnnnn nn 99.99 NNNN\\n\\nNnnn "..., 
    string2=0x1835980 "DESCRIPTION;LANGUAGE=en-US:Nn Nnnnn\\,\\n\\nNnnnnnnnn nnn nnn nnnnnn nn nnnnnn\n nnnnnnn nnnn nnnnnnnnn. N nnnn nnnnnnnnn nn nnn nnnnnnnn nnnnnnn nnn nn nn\n nn nn-nnnnnnn nn Nnnnnnn nn 99.99 NNNN\\n\\nNnnn "...) at regex.c:846
#1  0x00000000005ee090 in re_match_2_internal (bufp=0xd6d650 <searchbufs+5072>, 
    string1=0x1835980 "DESCRIPTION;LANGUAGE=en-US:Nn Nnnnn\\,\\n\\nNnnnnnnnn nnn nnn nnnnnn nn nnnnnn\n nnnnnnn nnnn nnnnnnnnn. N nnnn nnnnnnnnn nn nnn nnnnnnnn nnnnnnn nnn nn nn\n nn nn-nnnnnnn nn Nnnnnnn nn 99.99 NNNN\\n\\nNnnn "..., size1=0, 
    string2=0x1835980 "DESCRIPTION;LANGUAGE=en-US:Nn Nnnnn\\,\\n\\nNnnnnnnnn nnn nnn nnnnnn nn nnnnnn\n nnnnnnn nnnn nnnnnnnnn. N nnnn nnnnnnnnn nn nnn nnnnnnnn nnnnnnn nnn nn nn\n nn nn-nnnnnnn nn Nnnnnnn nn 99.99 NNNN\\n\\nNnnn "..., size2=40918, pos=0, regs=0xd6deb0 <search_regs>, stop=40918)
    at regex.c:5844
#2  0x00000000005e9022 in re_search_2 (bufp=0xd6d650 <searchbufs+5072>, 
    str1=0x1835980 "DESCRIPTION;LANGUAGE=en-US:Nn Nnnnn\\,\\n\\nNnnnnnnnn nnn nnn nnnnnn nn nnnnnn\n nnnnnnn nnnn nnnnnnnnn. N nnnn nnnnnnnnn nn nnn nnnnnnnn nnnnnnn nnn nn nn\n nn nn-nnnnnnn nn Nnnnnnn nn 99.99 NNNN\\n\\nNnnn "..., size1=0, 
    str2=0x1835980 "DESCRIPTION;LANGUAGE=en-US:Nn Nnnnn\\,\\n\\nNnnnnnnnn nnn nnn nnnnnn nn nnnnnn\n nnnnnnn nnnn nnnnnnnnn. N nnnn nnnnnnnnn nn nnn nnnnnnnn nnnnnnn nnn nn nn\n nn nn-nnnnnnn nn Nnnnnnn nn 99.99 NNNN\\n\\nNnnn "..., size2=40918, startpos=0, range=40918, regs=0xd6deb0 <search_regs>, 
    stop=40918) at regex.c:4470
#3  0x00000000005d6c06 in search_buffer (string=25301860, pos=1, pos_byte=1, lim=40891, 
    lim_byte=40919, n=1, RE=1, trt=20893029, inverse_trt=20483397, posix=false) at search.c:1265
#4  0x00000000005d63a1 in search_command (string=25301860, bound=0, noerror=44544, count=0, 
    direction=1, RE=1, posix=false) at search.c:1058
(More stack frames follow...)


(gdb) cont
[...]

PUSH_FAILURE_POINT:
  Before push, next avail: 5115
			size: 5120

  Push frame index: 5115
  Push string 0x1836013: ".nnn>\;\n> +NNNN <NNN..."
0:	/on_failure_jump to 7
3:	/anychar
4:	/jump to 0
7:	/stop_memory/1
9:	/jump to -8
12:	/succeed
13:	end of pattern.
  Push pattern 0x144aa8f: 
0x144aa92: EXECUTING anychar.
  Matched "46".

0x144aa93: EXECUTING jump -7 (to 0x144aa8f).

0x144aa8f: EXECUTING on_failure_jump 4 (to 0x144aa96):

PUSH_FAILURE_POINT:
  Before push, next avail: 5118
			size: 5120

Thread 1 "emacs" hit Hardware watchpoint 5: -location string1[0]

Old value = 68 'D'
New value = 0 '\000'
0x00007ffff0cc01a7 in __memset_sse2_unaligned_erms () from /usr/lib/libc.so.6

(gdb) bt 13
#0  0x00007ffff0cc01a7 in __memset_sse2_unaligned_erms () from /usr/lib/libc.so.6
#1  0x00000000006d27f5 in r_alloc_sbrk (size=290816) at ralloc.c:848
#2  0x00000000006ced96 in get_contiguous_space (size=290816, position=0x1833000) at gmalloc.c:476
#3  0x00000000006cf92a in _malloc_internal_nolock (size=163840) at gmalloc.c:844
#4  0x00000000006cfe9d in _malloc_internal (size=163840) at gmalloc.c:927
#5  0x00000000006cff1a in gmalloc (size=163840) at gmalloc.c:951
#6  0x00000000006d14e4 in malloc (size=163840) at gmalloc.c:1827
#7  0x00000000005f3e6b in lmalloc (size=163840) at alloc.c:1414
#8  0x00000000005f3356 in xmalloc (size=163840) at alloc.c:821
#9  0x00000000005f38e4 in record_xmalloc (size=163840) at alloc.c:1038
#10 0x00000000005ee233 in re_match_2_internal (bufp=0xd6d650 <searchbufs+5072>, string1=0x1835980 "", 
    size1=0, string2=0x1835980 "", size2=40918, pos=0, regs=0xd6deb0 <search_regs>, stop=40918)
    at regex.c:5844
#11 0x00000000005e9022 in re_search_2 (bufp=0xd6d650 <searchbufs+5072>, str1=0x1835980 "", size1=0, 
    str2=0x1835980 "", size2=40918, startpos=0, range=40918, regs=0xd6deb0 <search_regs>, stop=40918)
    at regex.c:4470
#12 0x00000000005d6c06 in search_buffer (string=25301860, pos=1, pos_byte=1, lim=40891, 
    lim_byte=40919, n=1, RE=1, trt=20893029, inverse_trt=20483397, posix=false) at search.c:1265
(More stack frames follow...)

Continuing.

Thread 1 "emacs" hit Hardware watchpoint 5: -location string1[0]

Old value = 0 '\000'
New value = -34 '\336'
0x00007ffff0d67b64 in __memcpy_ssse3 () from /usr/lib/libc.so.6

(gdb) cont
Continuing.

  Doubled stack; size now: 20480
	 slots available: 15362

  Push frame index: 5118
  Push string 0x1836014: "ª$..."

[...]

PUSH_FAILURE_POINT:
  Before push, next avail: 5130
			size: 20480

  Push frame index: 5130
  Push string 0x1836018: "ª$\..."
0:	/on_failure_jump to 7
3:	/anychar
4:	/jump to 0
7:	/stop_memory/1
9:	/jump to -8
12:	/succeed
13:	end of pattern.
  Push pattern 0x144aa8f: 
0x144aa92: EXECUTING anychar.

character.h:696: Emacs fatal error: assertion failed: CHAR_VALID_P (ch)

Thread 1 "emacs" hit Breakpoint 1, terminate_due_to_signal (sig=6, backtrace_limit=2147483647)
    at emacs.c:354
354	  signal (sig, SIG_DFL);

(gdb) bt 7
#0  terminate_due_to_signal (sig=6, backtrace_limit=2147483647) at emacs.c:354
#1  0x00000000005fdb9b in die (msg=0x725888 "CHAR_VALID_P (ch)", file=0x72587c "character.h", 
    line=696) at alloc.c:7224
#2  0x000000000056c000 in char_table_translate (obj=20893029, ch=4195178) at character.h:696
#3  0x00000000005eb8db in re_match_2_internal (bufp=0xd6d650 <searchbufs+5072>, 
    string1=0x1835980 "\336[\203\001", size1=0, string2=0x1835980 "\336[\203\001", size2=40918, 
    pos=0, regs=0xd6deb0 <search_regs>, stop=40918) at regex.c:5454
#4  0x00000000005e9022 in re_search_2 (bufp=0xd6d650 <searchbufs+5072>, 
    str1=0x1835980 "\336[\203\001", size1=0, str2=0x1835980 "\336[\203\001", size2=40918, startpos=0, 
    range=40918, regs=0xd6deb0 <search_regs>, stop=40918) at regex.c:4470
#5  0x00000000005d6c06 in search_buffer (string=25301860, pos=1, pos_byte=1, lim=40891, 
    lim_byte=40919, n=1, RE=1, trt=20893029, inverse_trt=20483397, posix=false) at search.c:1265
#6  0x00000000005d63a1 in search_command (string=25301860, bound=0, noerror=44544, count=0, 
    direction=1, RE=1, posix=false) at search.c:1058
(More stack frames follow...)

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: 24358@debbugs.gnu.org,  peder@klingenberg.no
> Date: Sat, 08 Oct 2016 09:45:20 -0400
> 
> >> From: npostavs@users.sourceforge.net
> >> Date: Fri, 07 Oct 2016 20:29:36 -0400
> >> Cc: 24358@debbugs.gnu.org
> >> 
> >> npostavs@users.sourceforge.net writes:
> >> >
> >> >> (I'm also on GNU/Linux, Arch) I get the same max-specpdl-size error with
> >> >> 25.1.50, with 24.5 (and below) I get (error "Stack overflow in regexp
> >> >> matcher")
> >> 
> >> icalendar--read-element has been fixed, but this still reproduces when
> >> doing (re-search-forward ".*\\(\n.*\\)*" nil t) on the text file given
> >> in the OP.
> >
> > Isn't that "user error"?
> 
> Yes, but it should give "Stack overflow in regexp matcher", not overflow
> the lisp stack (or assertion failure).

But that's what you said (see above): "Stack overflow in regexp
matcher".  That's what I meant when I said "user error".

> I show some more excerpts in the attached bug-24358-debug.log, but my
> main finding is that string1 of re_match_2_internal is originally:
> 
>     string1=0x1835980 "DESCRIPTION;LANGUAGE=
> 
> but then it becomes corrupted during a malloc:
> 
> Old value = 68 'D'
> New value = 0 '\000'

If that string is data of a Lisp string, then a call to malloc could
relocate the data.  Code that holds C pointers into buffer or string
text should either use SREF, or recompute the C pointer after each
function call which could GC.

Thanks.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> Date: Sat, 08 Oct 2016 17:39:10 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 24358@debbugs.gnu.org, peder@klingenberg.no
> 
> > Old value = 68 'D'
> > New value = 0 '\000'
> 
> If that string is data of a Lisp string, then a call to malloc could
> relocate the data.  Code that holds C pointers into buffer or string
> text should either use SREF, or recompute the C pointer after each
> function call which could GC.

Or maybe that string was already free'd?

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: npostavs@users.sourceforge.net
>> Cc: 24358@debbugs.gnu.org,  peder@klingenberg.no
>> Date: Sat, 08 Oct 2016 09:45:20 -0400
>> 
>> >> From: npostavs@users.sourceforge.net
>> >> Date: Fri, 07 Oct 2016 20:29:36 -0400
>> >> Cc: 24358@debbugs.gnu.org
>> >> 
>> >> npostavs@users.sourceforge.net writes:
>> >> >
>> >> >> (I'm also on GNU/Linux, Arch) I get the same max-specpdl-size error with
>> >> >> 25.1.50, with 24.5 (and below) I get (error "Stack overflow in regexp
>> >> >> matcher")
>> >> 
>> >> icalendar--read-element has been fixed, but this still reproduces when
>> >> doing (re-search-forward ".*\\(\n.*\\)*" nil t) on the text file given
>> >> in the OP.
>> >
>> > Isn't that "user error"?
>> 
>> Yes, but it should give "Stack overflow in regexp matcher", not overflow
>> the lisp stack (or assertion failure).
>
> But that's what you said (see above): "Stack overflow in regexp
> matcher".  That's what I meant when I said "user error".

Ah, I may have been a bit too terse there.  What I meant was, in Emacs
24.5 I correctly get "Stack overflow in regexp matcher", whereas in
emacs-master I get "Variable binding depth exceeds max-specpdl-size".
In emacs-25 I get the assertion failure.

>
>> I show some more excerpts in the attached bug-24358-debug.log, but my
>> main finding is that string1 of re_match_2_internal is originally:
>> 
>>     string1=0x1835980 "DESCRIPTION;LANGUAGE=
>> 
>> but then it becomes corrupted during a malloc:
>> 
>> Old value = 68 'D'
>> New value = 0 '\000'
>
> If that string is data of a Lisp string, then a call to malloc could
> relocate the data.  Code that holds C pointers into buffer or string
> text should either use SREF, or recompute the C pointer after each
> function call which could GC.

In that case, I believe the problem is that search_buffer calls
re_search_2 with a pointer to the buffer text, and then
re_match_2_internal (called by re_search_2), can allocate when doing
PUSH_FAILURE_POINT because it eventually does SAFE_ALLOCA to grow the
regex stack.

AFAICT, this bug is still present 24.5, but because re_max_failures is
set to a round number (see
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24358#27), there are fewer
calls to malloc and thus less chance of relocating the particular string
in question.

So possible solutions I can would be to pass down the lisp reference to
re_match_2_internal, or else set re_max_failures according to MAX_ALLOCA
(but this would make it much smaller).

search_buffer()
      /* Get pointers and sizes of the two strings
	 that make up the visible portion of the buffer. */

      p1 = BEGV_ADDR;
      s1 = GPT_BYTE - BEGV_BYTE;
      p2 = GAP_END_ADDR;
      s2 = ZV_BYTE - GPT_BYTE;

[...]

	  val = re_search_2 (bufp, (char *) p1, s1, (char *) p2, s2,
			     pos_byte - BEGV_BYTE, lim_byte - pos_byte,
			     (NILP (Vinhibit_changing_match_data)
			      ? &search_regs : &search_regs_1),
			     /* Don't allow match past current point */
			     pos_byte - BEGV_BYTE);

re_match_2_internal()
	case on_failure_jump:
	  EXTRACT_NUMBER_AND_INCR (mcnt, p);
	  DEBUG_PRINT ("EXECUTING on_failure_jump %d (to %p):\n",
		       mcnt, p + mcnt);

	  PUSH_FAILURE_POINT (p -3, d);

#define PUSH_FAILURE_POINT(pattern, string_place)
do { ...
  ENSURE_FAIL_STACK (NUM_NONREG_ITEMS);...

#define ENSURE_FAIL_STACK(space)					\
while (REMAINING_AVAIL_SLOTS <= space) {				\
  if (!GROW_FAIL_STACK (fail_stack))					\
    return -2;...

#define GROW_FAIL_STACK(fail_stack)					\
      ...
      = REGEX_REALLOCATE_STACK ((fail_stack).stack,			\
	  (fail_stack).size * sizeof (fail_stack_elt_t),		\
	  min (re_max_failures * TYPICAL_FAILURE_SIZE,			\
	       ((fail_stack).size * sizeof (fail_stack_elt_t)		\
		* FAIL_STACK_GROWTH_FACTOR))),				\

# define REGEX_ALLOCATE_STACK(size) REGEX_ALLOCATE (size)
# define REGEX_REALLOCATE_STACK(source, o, n) REGEX_REALLOCATE (source, o, n)

#  define REGEX_ALLOCATE SAFE_ALLOCA

/* SAFE_ALLOCA normally allocates memory on the stack, but if size is
   larger than MAX_ALLOCA, use xmalloc to avoid overflowing the stack.  */

enum MAX_ALLOCA { MAX_ALLOCA = 16 * 1024 };

#define SAFE_ALLOCA(size) ((size) < MAX_ALLOCA	\
			   ? alloca (size)	\
			   : (sa_must_free = true, record_xmalloc (size)))
                                       ^^^^^^^^^^^^^^^^^^^^^

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: 24358@debbugs.gnu.org,  peder@klingenberg.no
> Date: Sat, 08 Oct 2016 12:57:32 -0400
> 
> >> Old value = 68 'D'
> >> New value = 0 '\000'
> >
> > If that string is data of a Lisp string, then a call to malloc could
> > relocate the data.  Code that holds C pointers into buffer or string
> > text should either use SREF, or recompute the C pointer after each
> > function call which could GC.
> 
> In that case, I believe the problem is that search_buffer calls
> re_search_2 with a pointer to the buffer text, and then
> re_match_2_internal (called by re_search_2), can allocate when doing
> PUSH_FAILURE_POINT because it eventually does SAFE_ALLOCA to grow the
> regex stack.
> 
> AFAICT, this bug is still present 24.5, but because re_max_failures is
> set to a round number (see
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24358#27), there are fewer
> calls to malloc and thus less chance of relocating the particular string
> in question.
> 
> So possible solutions I can would be to pass down the lisp reference to
> re_match_2_internal, or else set re_max_failures according to MAX_ALLOCA
> (but this would make it much smaller).

Do you see buffer text actually changing its address in this scenario?
Otherwise, we might be chasing a wild goose.

(Once we establish this is the problem, we could talk about the fix.
IME, pointers could still be passed, but some extra caution is needed
when using them.)

Thanks.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

Eli Zaretskii <eliz@gnu.org> writes:

>
> Do you see buffer text actually changing its address in this scenario?
> Otherwise, we might be chasing a wild goose.

Yes, now that I know what to look for, it's easy enough.  I added just
this to .gdbinit:

    break search_command if (bound == 0)
    commands
      watch current_buffer->text->beg
      continue
    end
    # break only on the 2nd call to search_command
    ignore 3 1

And here is the result:

    Thread 1 "emacs" hit Breakpoint 3, search_command (string=25301860, bound=0, noerror=44544, count=0, 
        direction=1, RE=1, posix=false) at search.c:1024
    1024	  EMACS_INT n = direction;
    Hardware watchpoint 4: current_buffer->text->beg

    Thread 1 "emacs" hit Hardware watchpoint 4: current_buffer->text->beg

    Old value = (unsigned char *) 0x18351b8 ""
    New value = (unsigned char *) 0x188a1b8 ""
    r_alloc_sbrk (size=290816) at ralloc.c:818
    818		  for (b = last_bloc; b != NIL_BLOC; b = b->prev)
    (gdb) bt 12
    #0  r_alloc_sbrk (size=290816) at ralloc.c:818
    #1  0x00000000006ced96 in get_contiguous_space (size=290816, position=0x1833000) at gmalloc.c:476
    #2  0x00000000006cf92a in _malloc_internal_nolock (size=163840) at gmalloc.c:844
    #3  0x00000000006cfe9d in _malloc_internal (size=163840) at gmalloc.c:927
    #4  0x00000000006cff1a in gmalloc (size=163840) at gmalloc.c:951
    #5  0x00000000006d14e4 in malloc (size=163840) at gmalloc.c:1827
    #6  0x00000000005f3e6b in lmalloc (size=163840) at alloc.c:1414
    #7  0x00000000005f3356 in xmalloc (size=163840) at alloc.c:821
    #8  0x00000000005f38e4 in record_xmalloc (size=163840) at alloc.c:1038
    #9  0x00000000005ee233 in re_match_2_internal (bufp=0xd6d650 <searchbufs+5072>, 
        string1=0x1835988 "DESCRIPTION;LANGUAGE=en-US:Nn Nnnnn\\,\\n\\nNnnnnnnnn nnn nnn nnnnnn nn nnnnnn\n nnnnnnn nnnn nnnnnnnnn. N nnnn nnnnnnnnn nn nnn nnnnnnnn nnnnnnn nnn nn nn\n nn nn-nnnnnnn nn Nnnnnnn nn 99.99 NNNN\\n\\nNnnn "..., size1=0, 
        string2=0x1835988 "DESCRIPTION;LANGUAGE=en-US:Nn Nnnnn\\,\\n\\nNnnnnnnnn nnn nnn nnnnnn nn nnnnnn\n nnnnnnn nnnn nnnnnnnnn. N nnnn nnnnnnnnn nn nnn nnnnnnnn nnnnnnn nnn nn nn\n nn nn-nnnnnnn nn Nnnnnnn nn 99.99 NNNN\\n\\nNnnn "..., size2=40918, pos=0, regs=0xd6deb0 <search_regs>, stop=40918)
        at regex.c:5844
    #10 0x00000000005e9022 in re_search_2 (bufp=0xd6d650 <searchbufs+5072>, 
        str1=0x1835988 "DESCRIPTION;LANGUAGE=en-US:Nn Nnnnn\\,\\n\\nNnnnnnnnn nnn nnn nnnnnn nn nnnnnn\n nnnnnnn nnnn nnnnnnnnn. N nnnn nnnnnnnnn nn nnn nnnnnnnn nnnnnnn nnn nn nn\n nn nn-nnnnnnn nn Nnnnnnn nn 99.99 NNNN\\n\\nNnnn "..., size1=0, 
        str2=0x1835988 "DESCRIPTION;LANGUAGE=en-US:Nn Nnnnn\\,\\n\\nNnnnnnnnn nnn nnn nnnnnn nn nnnnnn\n nnnnnnn nnnn nnnnnnnnn. N nnnn nnnnnnnnn nn nnn nnnnnnnn nnnnnnn nnn nn nn\n nn nn-nnnnnnn nn Nnnnnnn nn 99.99 NNNN\\n\\nNnnn "..., size2=40918, startpos=0, range=40918, regs=0xd6deb0 <search_regs>, 
        stop=40918) at regex.c:4470
    #11 0x00000000005d6c06 in search_buffer (string=25301860, pos=1, pos_byte=1, lim=40891, 
        lim_byte=40919, n=1, RE=1, trt=20893029, inverse_trt=20483397, posix=false) at search.c:1265
    (More stack frames follow...)

    Lisp Backtrace:
    "re-search-forward" (0xffffc2e0)
    "progn" (0xffffc460)
    "unwind-protect" (0xffffc5a0)
    "save-current-buffer" (0xffffc710)
    "let" (0xffffc910)
    "eval-buffer" (0xffffcbf0)
    "load-with-code-conversion" (0xffffd158)
    "load" (0xffffd4e8)
    "command-line-1" (0xffffda30)
    "command-line" (0xffffdfe8)
    "normal-top-level" (0xffffe490)

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: 24358@debbugs.gnu.org,  peder@klingenberg.no
> Date: Sat, 08 Oct 2016 14:52:26 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >
> > Do you see buffer text actually changing its address in this scenario?
> > Otherwise, we might be chasing a wild goose.
> 
> Yes, now that I know what to look for, it's easy enough.  I added just
> this to .gdbinit:
> 
>     break search_command if (bound == 0)
>     commands
>       watch current_buffer->text->beg
>       continue
>     end
>     # break only on the 2nd call to search_command
>     ignore 3 1
> 
> And here is the result:
> 
>     Thread 1 "emacs" hit Breakpoint 3, search_command (string=25301860, bound=0, noerror=44544, count=0, 
>         direction=1, RE=1, posix=false) at search.c:1024
>     1024	  EMACS_INT n = direction;
>     Hardware watchpoint 4: current_buffer->text->beg
> 
>     Thread 1 "emacs" hit Hardware watchpoint 4: current_buffer->text->beg
> 
>     Old value = (unsigned char *) 0x18351b8 ""
>     New value = (unsigned char *) 0x188a1b8 ""
>     r_alloc_sbrk (size=290816) at ralloc.c:818

r_alloc_sbrk?  What OS is this?  We only use ralloc.c on a handful of
them, as of Emacs 25.

Anyway, the way to countermand this is to record in a local variable
the offset from beginning of buffer text to the value of the C pointer
before the call to record_xmalloc, then apply the offset after the
call to the new buffer text address.  (Let me know if this is clear
enough.)

You can find an example of this in coding.c:decode_coding_emacs_mule
(search for "relocated" in that function).

Thanks.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

Eli Zaretskii <eliz@gnu.org> writes:
>> 
>>     Thread 1 "emacs" hit Hardware watchpoint 4: current_buffer->text->beg
>> 
>>     Old value = (unsigned char *) 0x18351b8 ""
>>     New value = (unsigned char *) 0x188a1b8 ""
>>     r_alloc_sbrk (size=290816) at ralloc.c:818
>
> r_alloc_sbrk?  What OS is this?  We only use ralloc.c on a handful of
> them, as of Emacs 25.

Uh, it's GNU/Linux (Arch), so not too obscure I would think.  Where is
the decision to use ralloc made?  Maybe something went wrong in my
configure?

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: 24358@debbugs.gnu.org,  peder@klingenberg.no
> Date: Sat, 08 Oct 2016 16:55:17 -0400
> 
> > r_alloc_sbrk?  What OS is this?  We only use ralloc.c on a handful of
> > them, as of Emacs 25.
> 
> Uh, it's GNU/Linux (Arch), so not too obscure I would think.

I thought no GNU/Linux system uses ralloc.c, but maybe I was mistaken.

> Where is the decision to use ralloc made?

It was always a fallback, for systems that don't have better ways of
allocating system memory.  MS-Windows was using it until 25.1.

> Maybe something went wrong in my configure?

I guess configure somehow decides that gmalloc.c should be used on
your system (as opposed to the system-provided malloc), and then
ralloc.c is a necessity.  Look in your config.log for any test related
to malloc or something that has "malloc" as its substring.

In any case, this doesn't invalidate the bug, of course.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

I wrote:
>>>> So possible solutions I can would be to pass down the lisp reference to
>>>> re_match_2_internal, [...]
 
Eli Zaretskii <eliz@gnu.org> writes:
>>> (Once we establish this is the problem, we could talk about the fix.
>>> IME, pointers could still be passed, but some extra caution is needed
>>> when using them.)
[...]
> Anyway, the way to countermand this is to record in a local variable
> the offset from beginning of buffer text to the value of the C pointer
> before the call to record_xmalloc, then apply the offset after the
> call to the new buffer text address.  (Let me know if this is clear
> enough.)
>
> You can find an example of this in coding.c:decode_coding_emacs_mule
> (search for "relocated" in that function).

This does involve passing down the lisp reference, right?  Just want to
make sure I'm not missing something obvious before I start changing the
interface on a bunch of functions.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: 24358@debbugs.gnu.org,  peder@klingenberg.no
> Date: Wed, 12 Oct 2016 21:29:34 -0400
> 
> > Anyway, the way to countermand this is to record in a local variable
> > the offset from beginning of buffer text to the value of the C pointer
> > before the call to record_xmalloc, then apply the offset after the
> > call to the new buffer text address.  (Let me know if this is clear
> > enough.)
> >
> > You can find an example of this in coding.c:decode_coding_emacs_mule
> > (search for "relocated" in that function).
> 
> This does involve passing down the lisp reference, right?  Just want to
> make sure I'm not missing something obvious before I start changing the
> interface on a bunch of functions.

Aren't we talking about searching buffer text in this case?  If so,
the start address of the buffer text is known globally, it is given by
current_buffer->text->beg.  You just need to calculate the difference
between the start address before and after the call to malloc, or to a
function that might call malloc.

Alternatively (and more safely), save the value of PTR_BYTE_POS before
the call to malloc, and restore the C text pointer after malloc with
BYTE_POS_ADDR.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: npostavs@users.sourceforge.net
>> Cc: 24358@debbugs.gnu.org,  peder@klingenberg.no
>> Date: Wed, 12 Oct 2016 21:29:34 -0400
>> 
>> > Anyway, the way to countermand this is to record in a local variable
>> > the offset from beginning of buffer text to the value of the C pointer
>> > before the call to record_xmalloc, then apply the offset after the
>> > call to the new buffer text address.  (Let me know if this is clear
>> > enough.)
>> >
>> > You can find an example of this in coding.c:decode_coding_emacs_mule
>> > (search for "relocated" in that function).
>> 
>> This does involve passing down the lisp reference, right?  Just want to
>> make sure I'm not missing something obvious before I start changing the
>> interface on a bunch of functions.
>
> Aren't we talking about searching buffer text in this case?  If so,
> the start address of the buffer text is known globally, it is given by
> current_buffer->text->beg.

The particular crash reported is during a buffer search, but the bug is
in re_match_2_internal which (if I understand correctly) may be called
for string search as well.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> X-Spam-Status: No, score=0.0 required=5.0 tests=BAYES_40,FREEMAIL_FROM,
> 	T_DKIM_INVALID autolearn=disabled version=3.3.2
> From: npostavs@users.sourceforge.net
> Cc: 24358@debbugs.gnu.org,  peder@klingenberg.no
> Date: Thu, 13 Oct 2016 22:19:09 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: npostavs@users.sourceforge.net
> >> Cc: 24358@debbugs.gnu.org,  peder@klingenberg.no
> >> Date: Wed, 12 Oct 2016 21:29:34 -0400
> >> 
> >> > Anyway, the way to countermand this is to record in a local variable
> >> > the offset from beginning of buffer text to the value of the C pointer
> >> > before the call to record_xmalloc, then apply the offset after the
> >> > call to the new buffer text address.  (Let me know if this is clear
> >> > enough.)
> >> >
> >> > You can find an example of this in coding.c:decode_coding_emacs_mule
> >> > (search for "relocated" in that function).
> >> 
> >> This does involve passing down the lisp reference, right?  Just want to
> >> make sure I'm not missing something obvious before I start changing the
> >> interface on a bunch of functions.
> >
> > Aren't we talking about searching buffer text in this case?  If so,
> > the start address of the buffer text is known globally, it is given by
> > current_buffer->text->beg.
> 
> The particular crash reported is during a buffer search, but the bug is
> in re_match_2_internal which (if I understand correctly) may be called
> for string search as well.

Then yes, you will need to somehow pass down the object from which the
text comes.  It can be in some global variable, for instance, if
changing the function's signature is undesirable.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

[-- Attachment #1: Type: text/plain, Size: 813 bytes --]

tags 24358 patch
quit

Eli Zaretskii <eliz@gnu.org> writes:
>
> Then yes, you will need to somehow pass down the object from which the
> text comes.  It can be in some global variable, for instance, if
> changing the function's signature is undesirable.

I decided that a global variable would be a bad idea since it would
still effectively be part of the calling convention, and too easy for
callers to forget.  But just after finishing the patch I noticed this:

    /* In Emacs, this is the string or buffer in which we
       are matching.  It is used for looking up syntax properties.  */
    Lisp_Object re_match_object;

So now I'm thinking it might be better to reuse that variable instead.
Although it only seems to get to Qt, Qnil, and string objects; I'm not
sure what the meaning of Qt and Qnil are.


[-- Attachment #2: patch v1 --]
[-- Type: text/plain, Size: 16796 bytes --]

From 1016c78fe51183ed4d89eeb90c1c06f43c23cbb8 Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Mon, 17 Oct 2016 22:17:27 -0400
Subject: [PATCH v1] Fix handling of allocation in regex matching

`re_match_2_internal' uses pointers to the lisp objects that it
searches.  Since it may call malloc when growing the "fail stack", these
pointers may be invalidated while searching, resulting in memory
curruption (Bug #24358).

To fix this, we check the pointer that the lisp object points to before
and after growing the stack, and update existing pointers accordingly.
This means that all callers of regex searching functions must pass a
reference to the lisp object that they search.  Callers searching pure C
strings that can't relocate pass Qnil.  To reduce the need for
preprocessor conditionals, we define Lisp_Object as an enum with just
the value Qnil when building regex.c for non-emacs programs (etags).

* src/regex.c (STR_BASE_PTR): New macro.
(ENSURE_FAIL_STACK): Use it to update pointers after growing the stack.
(re_search, re_search_2, re_match_2, re_match_2_internal): Add BASE
parameter.
* src/dired.c (directory_files_internal):
* src/search.c (looking_at_1, string_match_1):
(fast_string_match_internal, fast_looking_at, search_buffer): Pass the
searched lisp object to re_search, re_search_2, re_match_2 as the BASE
parameter.
* src/search.c (fast_c_string_match_ignore_case): Pass Qnil as BASE
parameter.

* lib-src/etags.c (regex_tag_multiline):
* src/regex.c (re_match, regexec) [!emacs]: Pass dummy NO_LISP arg for
BASE.
* src/regex.h (Lisp_Object) [!emacs]: New single valued enum.
---
 lib-src/etags.c |  2 +-
 src/dired.c     |  2 +-
 src/regex.c     | 79 +++++++++++++++++++++++++++++++++++++++++----------------
 src/regex.h     | 12 ++++++++-
 src/search.c    | 45 +++++++++++++++++++-------------
 5 files changed, 98 insertions(+), 42 deletions(-)

diff --git a/lib-src/etags.c b/lib-src/etags.c
index 1457700..c8fffc2 100644
--- a/lib-src/etags.c
+++ b/lib-src/etags.c
@@ -6304,7 +6304,7 @@ regex_tag_multiline (void)
 
       while (match >= 0 && match < filebuf.len)
 	{
-	  match = re_search (rp->pat, buffer, filebuf.len, charno,
+	  match = re_search (rp->pat, NO_LISP, buffer, filebuf.len, charno,
 			     filebuf.len - match, &rp->regs);
 	  switch (match)
 	    {
diff --git a/src/dired.c b/src/dired.c
index dba575c..a558aa2 100644
--- a/src/dired.c
+++ b/src/dired.c
@@ -259,7 +259,7 @@ directory_files_internal (Lisp_Object directory, Lisp_Object full,
       QUIT;
 
       bool wanted = (NILP (match)
-		     || re_search (bufp, SSDATA (name), len, 0, len, 0) >= 0);
+		     || re_search (bufp, name, SSDATA (name), len, 0, len, 0) >= 0);
 
       immediate_quit = 0;
 
diff --git a/src/regex.c b/src/regex.c
index 164eb46..659b1f9 100644
--- a/src/regex.c
+++ b/src/regex.c
@@ -533,12 +533,11 @@ init_syntax_once (void)
 
 typedef char boolean;
 
-static regoff_t re_match_2_internal (struct re_pattern_buffer *bufp,
-				     re_char *string1, size_t size1,
-				     re_char *string2, size_t size2,
-				     ssize_t pos,
-				     struct re_registers *regs,
-				     ssize_t stop);
+static regoff_t
+re_match_2_internal (struct re_pattern_buffer *bufp, Lisp_Object string_base,
+                     const_re_char *string1, size_t size1,
+                     const_re_char *string2, size_t size2,
+                     ssize_t pos, struct re_registers *regs, ssize_t stop);
 \f
 /* These are the command codes that appear in compiled regular
    expressions.  Some opcodes are followed by argument bytes.  A
@@ -1436,11 +1435,38 @@ WEAK_ALIAS (__re_set_syntax, re_set_syntax)
 #define NEXT_FAILURE_HANDLE(h) fail_stack.stack[(h) - 3].integer
 #define TOP_FAILURE_HANDLE() fail_stack.frame
 
+#ifdef emacs
+#define STR_BASE_PTR(obj)                       \
+    (BUFFERP (obj)? XBUFFER (obj)->text->beg :  \
+     STRINGP (obj)? SDATA (obj) :               \
+     NULL)
+#else
+#define STR_BASE_PTR(obj) ((re_char*)0)
+#endif
 
 #define ENSURE_FAIL_STACK(space)					\
 while (REMAINING_AVAIL_SLOTS <= space) {				\
+  re_char* orig_base = STR_BASE_PTR (string_base);                      \
   if (!GROW_FAIL_STACK (fail_stack))					\
-    return -2;								\
+    return -2;                                                          \
+  /* GROW_FAIL_STACK may call malloc and relocate the string */         \
+  /* pointers.  */                                                      \
+  ptrdiff_t delta = STR_BASE_PTR (string_base) - orig_base;             \
+  if (string1)                                                          \
+    {                                                                   \
+      string1 += delta;                                                 \
+      end1 += delta;                                                    \
+      end_match_1 += delta;                                             \
+    }                                                                   \
+  if (string2)                                                          \
+    {                                                                   \
+      string2 += delta;                                                 \
+      end2 += delta;                                                    \
+      end_match_2 += delta;                                             \
+    }                                                                   \
+  d += delta;                                                           \
+  dend += delta;                                                        \
+  dfail += delta;                                                       \
   DEBUG_PRINT ("\n  Doubled stack; size now: %zd\n", (fail_stack).size);\
   DEBUG_PRINT ("	 slots available: %zd\n", REMAINING_AVAIL_SLOTS);\
 }
@@ -4222,10 +4248,11 @@ WEAK_ALIAS (__re_set_registers, re_set_registers)
    doesn't let you say where to stop matching. */
 
 regoff_t
-re_search (struct re_pattern_buffer *bufp, const char *string, size_t size,
+re_search (struct re_pattern_buffer *bufp,
+           Lisp_Object base, const char *string, size_t size,
 	   ssize_t startpos, ssize_t range, struct re_registers *regs)
 {
-  return re_search_2 (bufp, NULL, 0, string, size, startpos, range,
+  return re_search_2 (bufp, base, NULL, 0, string, size, startpos, range,
 		      regs, size);
 }
 WEAK_ALIAS (__re_search, re_search)
@@ -4260,8 +4287,10 @@ WEAK_ALIAS (__re_search, re_search)
    stack overflow).  */
 
 regoff_t
-re_search_2 (struct re_pattern_buffer *bufp, const char *str1, size_t size1,
-	     const char *str2, size_t size2, ssize_t startpos, ssize_t range,
+re_search_2 (struct re_pattern_buffer *bufp, Lisp_Object str_base,
+             const char *str1, size_t size1,
+             const char *str2, size_t size2,
+             ssize_t startpos, ssize_t range,
 	     struct re_registers *regs, ssize_t stop)
 {
   regoff_t val;
@@ -4443,7 +4472,8 @@ re_search_2 (struct re_pattern_buffer *bufp, const char *str1, size_t size1,
 	  && !bufp->can_be_null)
 	return -1;
 
-      val = re_match_2_internal (bufp, string1, size1, string2, size2,
+      val = re_match_2_internal (bufp, str_base,
+                                 string1, size1, string2, size2,
 				 startpos, regs, stop);
 
       if (val >= 0)
@@ -4879,8 +4909,10 @@ mutually_exclusive_p (struct re_pattern_buffer *bufp, const_re_char *p1,
 re_match (struct re_pattern_buffer *bufp, const char *string,
 	  size_t size, ssize_t pos, struct re_registers *regs)
 {
-  regoff_t result = re_match_2_internal (bufp, NULL, 0, (re_char*) string,
-					 size, pos, regs, size);
+  regoff_t result = re_match_2_internal (bufp, NO_LISP,
+                                         NULL, 0,
+                                         (re_char*) string, size,
+                                         pos, regs, size);
   return result;
 }
 WEAK_ALIAS (__re_match, re_match)
@@ -4906,9 +4938,10 @@ WEAK_ALIAS (__re_match, re_match)
    matched substring.  */
 
 regoff_t
-re_match_2 (struct re_pattern_buffer *bufp, const char *string1,
-	    size_t size1, const char *string2, size_t size2, ssize_t pos,
-	    struct re_registers *regs, ssize_t stop)
+re_match_2 (struct re_pattern_buffer *bufp, Lisp_Object base,
+            const char *string1, size_t size1,
+            const char *string2, size_t size2,
+            ssize_t pos, struct re_registers *regs, ssize_t stop)
 {
   regoff_t result;
 
@@ -4919,8 +4952,9 @@ re_match_2 (struct re_pattern_buffer *bufp, const char *string1,
   SETUP_SYNTAX_TABLE_FOR_OBJECT (re_match_object, charpos, 1);
 #endif
 
-  result = re_match_2_internal (bufp, (re_char*) string1, size1,
-				(re_char*) string2, size2,
+  result = re_match_2_internal (bufp, base,
+                                (re_char*) string1, size1,
+                                (re_char*) string2, size2,
 				pos, regs, stop);
   return result;
 }
@@ -4930,8 +4964,9 @@ WEAK_ALIAS (__re_match_2, re_match_2)
 /* This is a separate function so that we can force an alloca cleanup
    afterwards.  */
 static regoff_t
-re_match_2_internal (struct re_pattern_buffer *bufp, const_re_char *string1,
-		     size_t size1, const_re_char *string2, size_t size2,
+re_match_2_internal (struct re_pattern_buffer *bufp, Lisp_Object string_base,
+                     const_re_char *string1, size_t size1,
+                     const_re_char *string2, size_t size2,
 		     ssize_t pos, struct re_registers *regs, ssize_t stop)
 {
   /* General temporaries.  */
@@ -6572,7 +6607,7 @@ regexec (const regex_t *_Restrict_ preg, const char *_Restrict_ string,
      by '\n' which would throw things off.  */
 
   /* Perform the searching operation.  */
-  ret = re_search (&private_preg, string, len,
+  ret = re_search (&private_preg, NO_LISP, string, len,
 		   /* start: */ 0, /* range: */ len,
 		   want_reg_info ? &regs : 0);
 
diff --git a/src/regex.h b/src/regex.h
index 817167a..4810bc4 100644
--- a/src/regex.h
+++ b/src/regex.h
@@ -469,13 +469,21 @@ extern const char *re_compile_pattern (const char *__pattern, size_t __length,
    internal error.  */
 extern int re_compile_fastmap (struct re_pattern_buffer *__buffer);
 
+#ifndef emacs
+/* Define Lisp_Object outside of emacs, just so something can be
+   passed as the BASE parameter to re_search and re_match.  */
+typedef enum { NO_LISP } Lisp_Object;
+#endif
 
 /* Search in the string STRING (with length LENGTH) for the pattern
    compiled into BUFFER.  Start searching at position START, for RANGE
    characters.  Return the starting position of the match, -1 for no
    match, or -2 for an internal error.  Also return register
-   information in REGS (if REGS and BUFFER->no_sub are nonzero).  */
+   information in REGS (if REGS and BUFFER->no_sub are nonzero).  If
+   STRING is a pointer into a lisp object, pass the object as BASE in
+   order to correctly handle relocation if re_search calls malloc.  */
 extern regoff_t re_search (struct re_pattern_buffer *__buffer,
+                           Lisp_Object __base,
 			   const char *__string, size_t __length,
 			   ssize_t __start, ssize_t __range,
 			   struct re_registers *__regs);
@@ -484,6 +492,7 @@ extern regoff_t re_search (struct re_pattern_buffer *__buffer,
 /* Like `re_search', but search in the concatenation of STRING1 and
    STRING2.  Also, stop searching at index START + STOP.  */
 extern regoff_t re_search_2 (struct re_pattern_buffer *__buffer,
+                             Lisp_Object __base,
 			     const char *__string1, size_t __length1,
 			     const char *__string2, size_t __length2,
 			     ssize_t __start, ssize_t __range,
@@ -500,6 +509,7 @@ extern regoff_t re_match (struct re_pattern_buffer *__buffer,
 
 /* Relates to `re_match' as `re_search_2' relates to `re_search'.  */
 extern regoff_t re_match_2 (struct re_pattern_buffer *__buffer,
+                            Lisp_Object __base,
 			    const char *__string1, size_t __length1,
 			    const char *__string2, size_t __length2,
 			    ssize_t __start, struct re_registers *__regs,
diff --git a/src/search.c b/src/search.c
index dc7e2d8..3d3d355 100644
--- a/src/search.c
+++ b/src/search.c
@@ -287,8 +287,10 @@ looking_at_1 (Lisp_Object string, bool posix)
   immediate_quit = 1;
   QUIT;			/* Do a pending quit right away, to avoid paradoxical behavior */
 
-  /* Get pointers and sizes of the two strings
-     that make up the visible portion of the buffer. */
+  /* Get pointers and sizes of the two strings that make up the
+     visible portion of the buffer.  Note that we can use pointers
+     here, unlike in search_buffer, because we only call re_match_2
+     once.  */
 
   p1 = BEGV_ADDR;
   s1 = GPT_BYTE - BEGV_BYTE;
@@ -308,7 +310,8 @@ looking_at_1 (Lisp_Object string, bool posix)
 
   re_match_object = Qnil;
 
-  i = re_match_2 (bufp, (char *) p1, s1, (char *) p2, s2,
+  i = re_match_2 (bufp, Fcurrent_buffer (),
+                  (char *) p1, s1, (char *) p2, s2,
 		  PT_BYTE - BEGV_BYTE,
 		  (NILP (Vinhibit_changing_match_data)
 		   ? &search_regs : NULL),
@@ -401,7 +404,7 @@ string_match_1 (Lisp_Object regexp, Lisp_Object string, Lisp_Object start,
   immediate_quit = 1;
   re_match_object = string;
 
-  val = re_search (bufp, SSDATA (string),
+  val = re_search (bufp, string, SSDATA (string),
 		   SBYTES (string), pos_byte,
 		   SBYTES (string) - pos_byte,
 		   (NILP (Vinhibit_changing_match_data)
@@ -473,7 +476,7 @@ fast_string_match_internal (Lisp_Object regexp, Lisp_Object string,
   immediate_quit = 1;
   re_match_object = string;
 
-  val = re_search (bufp, SSDATA (string),
+  val = re_search (bufp, string, SSDATA (string),
 		   SBYTES (string), 0,
 		   SBYTES (string), 0);
   immediate_quit = 0;
@@ -498,7 +501,7 @@ fast_c_string_match_ignore_case (Lisp_Object regexp,
 			  Vascii_canon_table, 0,
 			  0);
   immediate_quit = 1;
-  val = re_search (bufp, string, len, 0, len, 0);
+  val = re_search (bufp, Qnil, string, len, 0, len, 0);
   immediate_quit = 0;
   return val;
 }
@@ -561,7 +564,8 @@ fast_looking_at (Lisp_Object regexp, ptrdiff_t pos, ptrdiff_t pos_byte,
 
   buf = compile_pattern (regexp, 0, Qnil, 0, multibyte);
   immediate_quit = 1;
-  len = re_match_2 (buf, (char *) p1, s1, (char *) p2, s2,
+  len = re_match_2 (buf, Fcurrent_buffer (),
+                    (char *) p1, s1, (char *) p2, s2,
 		    pos_byte, NULL, limit_byte);
   immediate_quit = 0;
 
@@ -1178,8 +1182,8 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 
   if (RE && !(trivial_regexp_p (string) && NILP (Vsearch_spaces_regexp)))
     {
-      unsigned char *p1, *p2;
-      ptrdiff_t s1, s2;
+      unsigned char *base;
+      ptrdiff_t off1, off2, s1, s2;
       struct re_pattern_buffer *bufp;
 
       bufp = compile_pattern (string,
@@ -1193,16 +1197,19 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 				   can take too long. */
       QUIT;			/* Do a pending quit right away,
 				   to avoid paradoxical behavior */
-      /* Get pointers and sizes of the two strings
-	 that make up the visible portion of the buffer. */
+      /* Get offsets and sizes of the two strings that make up the
+         visible portion of the buffer.  We compute offsets instead of
+         pointers because re_search_2 may call malloc and therefore
+         change the buffer text address.  */
 
-      p1 = BEGV_ADDR;
+      base = current_buffer->text->beg;
+      off1 = BEGV_ADDR - base;
       s1 = GPT_BYTE - BEGV_BYTE;
-      p2 = GAP_END_ADDR;
+      off2 = GAP_END_ADDR - base;
       s2 = ZV_BYTE - GPT_BYTE;
       if (s1 < 0)
 	{
-	  p2 = p1;
+          off2 = off1;
 	  s2 = ZV_BYTE - BEGV_BYTE;
 	  s1 = 0;
 	}
@@ -1217,7 +1224,9 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 	{
 	  ptrdiff_t val;
 
-	  val = re_search_2 (bufp, (char *) p1, s1, (char *) p2, s2,
+          val = re_search_2 (bufp, Fcurrent_buffer (),
+                             (char*) (base + off1), s1,
+                             (char*) (base + off2), s2,
 			     pos_byte - BEGV_BYTE, lim_byte - pos_byte,
 			     (NILP (Vinhibit_changing_match_data)
 			      ? &search_regs : &search_regs_1),
@@ -1262,8 +1271,10 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 	{
 	  ptrdiff_t val;
 
-	  val = re_search_2 (bufp, (char *) p1, s1, (char *) p2, s2,
-			     pos_byte - BEGV_BYTE, lim_byte - pos_byte,
+          val = re_search_2 (bufp, Fcurrent_buffer (),
+                             (char*) (base + off1), s1,
+                             (char*) (base + off2), s2,
+                             pos_byte - BEGV_BYTE, lim_byte - pos_byte,
 			     (NILP (Vinhibit_changing_match_data)
 			      ? &search_regs : &search_regs_1),
 			     lim_byte - BEGV_BYTE);
-- 
2.9.3


[-- Attachment #3: Type: text/plain, Size: 1003 bytes --]



Sam Halliday <sam.halliday@gmail.com> writes:
>
> Could somebody please let me know how to dig into 0x56e5508 (if
> relevant)? I have never used gdb in anger.

See if you can adapt the .gdbinit modifications I posted in
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24358#53; assuming your
recipe always triggers the crash in the exact same way, it might be
enough just to change the conditional to 'if (lim == 1156)', and you
might not even need the ignore call at all.

> I've attempted to workaround this in ensime by simplifying our loading
> of the .ensime file (the s-expression) to use with-temp-buffer and not
> declaring .ensime as an emacs-lisp file... it may be that one of my
> minor modes, active in emacs-lisp, is performing a regexp that is
> triggering this bug.

If you can get the lisp backtrace it should show you what code is
triggering the regexp search.  It should show up automatically when you
do bt, if you've sourced src/.gdbinit as described in etc/DEBUG
"Configuring GDB".

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: 24358@debbugs.gnu.org, Sam Halliday <sam.halliday@gmail.com>
> Date: Tue, 18 Oct 2016 23:11:45 -0400
> 
> > Then yes, you will need to somehow pass down the object from which the
> > text comes.  It can be in some global variable, for instance, if
> > changing the function's signature is undesirable.
> 
> I decided that a global variable would be a bad idea since it would
> still effectively be part of the calling convention, and too easy for
> callers to forget.

That's true, but doing this is a standard Emacs coding practice, used
in quite a few other places.

> But just after finishing the patch I noticed this:
> 
>     /* In Emacs, this is the string or buffer in which we
>        are matching.  It is used for looking up syntax properties.  */
>     Lisp_Object re_match_object;

Indeed.

> So now I'm thinking it might be better to reuse that variable instead.
> Although it only seems to get to Qt, Qnil, and string objects; I'm not
> sure what the meaning of Qt and Qnil are.

nil means current buffer, t means a C string.  (This is standard Emacs
convention, used in other places as well, but you can verify it is
used in this case by looking at all the places where these values are
assigned.)  The last case, of a C string, doesn't need to bother about
relocation, of course (but you could ignore that distinction for
simplicity of the code, if you want, it will never do any harm).

I guess you will be rewriting your patch to use re_match_object?  I
expect it to be much simpler and smaller.  re_match_object is already
staticpro'd, btw, so you don't need to worry about GC for the Lisp
object (a string) that gets put in its value.  However, I think we
should assign Qnil to re_match_object as soon as re_match_2 returns,
to avoid having the string protected from GC for too long.

A couple of comments to the patch:

> +#ifdef emacs
> +#define STR_BASE_PTR(obj)                       \
> +    (BUFFERP (obj)? XBUFFER (obj)->text->beg :  \
> +     STRINGP (obj)? SDATA (obj) :               \
> +     NULL)

There's an unnecessary complication here: since, when regex functions
are invoked to search a buffer, that buffer is always the current
buffer, you don't need to pass the buffer object and use XBUFFER,
because the current buffer is always available in the global variable
current_buffer (which is a C struct, not a Lisp object).  So you could
adopt the usual semantics of passing Qnil to mean the current buffer
and a string object to mean that string.  Then the only test in the
macro should be STRINGP.

>  #define ENSURE_FAIL_STACK(space)					\
>  while (REMAINING_AVAIL_SLOTS <= space) {				\
> +  re_char* orig_base = STR_BASE_PTR (string_base);                      \
>    if (!GROW_FAIL_STACK (fail_stack))					\
> -    return -2;								\
> +    return -2;                                                          \
> +  /* GROW_FAIL_STACK may call malloc and relocate the string */         \
> +  /* pointers.  */                                                      \
> +  ptrdiff_t delta = STR_BASE_PTR (string_base) - orig_base;             \
> +  if (string1)                                                          \
> +    {                                                                   \
> +      string1 += delta;                                                 \
> +      end1 += delta;                                                    \
> +      end_match_1 += delta;                                             \
> +    }                                                                   \
> +  if (string2)                                                          \
> +    {                                                                   \
> +      string2 += delta;                                                 \
> +      end2 += delta;                                                    \
> +      end_match_2 += delta;                                             \
> +    }                                                                   \
> +  d += delta;                                                           \
> +  dend += delta;                                                        \
> +  dfail += delta;                                                       \

I think doing this via buffer and string positions instead would yield
slightly more portable code, since Standard C says subtraction of
pointers to different objects yields undefined behavior.

Thanks.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: npostavs@users.sourceforge.net
>> Cc: 24358@debbugs.gnu.org, Sam Halliday <sam.halliday@gmail.com>
>> Date: Tue, 18 Oct 2016 23:11:45 -0400
>> 
>>     /* In Emacs, this is the string or buffer in which we
>>        are matching.  It is used for looking up syntax properties.  */
>>     Lisp_Object re_match_object;
>
> Indeed.
>
>> So now I'm thinking it might be better to reuse that variable instead.
>> Although it only seems to get to Qt, Qnil, and string objects; I'm not
>> sure what the meaning of Qt and Qnil are.
>
> nil means current buffer, t means a C string.  (This is standard Emacs
> convention, used in other places as well, but you can verify it is
> used in this case by looking at all the places where these values are
> assigned.)
>

It would be nice to have a comment in the code about this.  I saw that
it was set to nil in search_buffer etc, but that just confused me since I
took nil to mean "nothing".

>
> I guess you will be rewriting your patch to use re_match_object?  I
> expect it to be much simpler and smaller.  re_match_object is already
> staticpro'd, btw, so you don't need to worry about GC for the Lisp
> object (a string) that gets put in its value.  However, I think we
> should assign Qnil to re_match_object as soon as re_match_2 returns,
> to avoid having the string protected from GC for too long.

Do you mean staticpro prevents relocation (not just collection)?
In that case wouldn't it be even simpler to assign the buffer object to
re_match_object?

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: 24358@debbugs.gnu.org,  sam.halliday@gmail.com
> Date: Wed, 19 Oct 2016 08:29:48 -0400
> 
> > nil means current buffer, t means a C string.  (This is standard Emacs
> > convention, used in other places as well, but you can verify it is
> > used in this case by looking at all the places where these values are
> > assigned.)
> 
> It would be nice to have a comment in the code about this.

Done.  (Please update the comment when you are done with the change of
using that variable for this additional purpose.)

> I saw that it was set to nil in search_buffer etc, but that just
> confused me since I took nil to mean "nothing".

Yeah, it can be confusing when you first bump into it.

> > I guess you will be rewriting your patch to use re_match_object?  I
> > expect it to be much simpler and smaller.  re_match_object is already
> > staticpro'd, btw, so you don't need to worry about GC for the Lisp
> > object (a string) that gets put in its value.  However, I think we
> > should assign Qnil to re_match_object as soon as re_match_2 returns,
> > to avoid having the string protected from GC for too long.
> 
> Do you mean staticpro prevents relocation (not just collection)?
> In that case wouldn't it be even simpler to assign the buffer object to
> re_match_object?

No, it only makes sure the objects will not be GC'ed, it doesn't
prevent relocation.  Every staticpro'd variable is marked at the
beginning of GC, which includes marking any Lisp object that is
referenced, directly or indirectly, from the staticpro'd variable.

Btw, note that regex.c already has macros PTR_TO_OFFSET and
POS_AS_IN_BUFFER which you can use.

Thanks.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

[-- Attachment #1: Type: text/plain, Size: 899 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:
>
>> +#ifdef emacs
>> +#define STR_BASE_PTR(obj)                       \
>> +    (BUFFERP (obj)? XBUFFER (obj)->text->beg :  \
>> +     STRINGP (obj)? SDATA (obj) :               \
>> +     NULL)
>

[...]

> the only test in the macro should be STRINGP.
>

Hmm, not sure I feel comfortable being that implicit.  I kept this macro
the same except for using (NILP (obj)? current_buffer->...) instead of
BUFFERP and XBUFFER.

>
> Btw, note that regex.c already has macros PTR_TO_OFFSET and
> POS_AS_IN_BUFFER which you can use.

AFAICT these are not useful for this: they give offets relative to
string1 (or string2), which would not help to compute the new value for
string1 (or string2, etc...).  Since I was looking at it, I've also
added a comment about the trick of punning the boolean result into
buffer or string base index.

------

Here is the new patch:


[-- Attachment #2: patch v2 --]
[-- Type: text/plain, Size: 12538 bytes --]

From 92700753ac38b947dd2a725478b07dd7ef229c3a Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Wed, 19 Oct 2016 20:23:50 -0400
Subject: [PATCH v2] Fix handling of allocation in regex matching

`re_match_2_internal' uses pointers to the lisp objects that it
searches.  Since it may call malloc when growing the "fail stack", these
pointers may be invalidated while searching, resulting in memory
curruption (Bug #24358).

To fix this, we check the pointer that the lisp object (as specified by
re_match_object) points to before and after growing the stack, and
update existing pointers accordingly.

* src/regex.c (STR_BASE_PTR): New macro.
(ENSURE_FAIL_STACK, re_search_2): Use it to convert pointers into
offsets before possible malloc call, and back into pointers again
afterwards.
(POS_AS_IN_BUFFER): Add explanatory comment about punning trick.
* src/search.c (search_buffer): Instead of storing search location as
pointers, store them as pointers and recompute the corresponding address
for each call to `re_search_2'.
(string_match_1, fast_string_match_internal, fast_looking_at):
* src/dired.c (directory_files_internal): Set `re_match_object' to Qnil
after calling `re_search' or `re_match_2'.
* src/regex.h (re_match_object): Mention new usage in commentary.
---
 src/dired.c  |  4 +++-
 src/regex.c  | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++---
 src/regex.h  |  3 ++-
 src/search.c | 36 ++++++++++++++++++----------
 4 files changed, 102 insertions(+), 17 deletions(-)

diff --git a/src/dired.c b/src/dired.c
index dba575c..006f74c 100644
--- a/src/dired.c
+++ b/src/dired.c
@@ -259,9 +259,11 @@ directory_files_internal (Lisp_Object directory, Lisp_Object full,
       QUIT;
 
       bool wanted = (NILP (match)
-		     || re_search (bufp, SSDATA (name), len, 0, len, 0) >= 0);
+		     || (re_match_object = name,
+                         re_search (bufp, SSDATA (name), len, 0, len, 0) >= 0));
 
       immediate_quit = 0;
+      re_match_object = Qnil;   /* Stop protecting name from GC.  */
 
       if (wanted)
 	{
diff --git a/src/regex.c b/src/regex.c
index 164eb46..b710f50 100644
--- a/src/regex.c
+++ b/src/regex.c
@@ -152,6 +152,8 @@
 
 /* Converts the pointer to the char to BEG-based offset from the start.  */
 # define PTR_TO_OFFSET(d) POS_AS_IN_BUFFER (POINTER_TO_OFFSET (d))
+/* Strings are 0-indexed, buffers are 1-indexed; we pun on the boolean
+   result to get the right base index.  */
 # define POS_AS_IN_BUFFER(p) ((p) + (NILP (re_match_object) || BUFFERP (re_match_object)))
 
 # define RE_MULTIBYTE_P(bufp) ((bufp)->multibyte)
@@ -1436,11 +1438,62 @@ WEAK_ALIAS (__re_set_syntax, re_set_syntax)
 #define NEXT_FAILURE_HANDLE(h) fail_stack.stack[(h) - 3].integer
 #define TOP_FAILURE_HANDLE() fail_stack.frame
 
+#ifdef emacs
+#define STR_BASE_PTR(obj)                   \
+  (NILP(obj)? current_buffer->text->beg :   \
+   STRINGP (obj)? SDATA (obj) :             \
+   NULL)
+#else
+#define STR_BASE_PTR(obj) NULL
+#endif
 
 #define ENSURE_FAIL_STACK(space)					\
 while (REMAINING_AVAIL_SLOTS <= space) {				\
+  re_char* orig_base = STR_BASE_PTR (re_match_object);                  \
+  ptrdiff_t string1_off, end1_off, end_match_1_off;                     \
+  ptrdiff_t string2_off, end2_off, end_match_2_off;                     \
+  ptrdiff_t d_off, dend_off, dfail_off;                                 \
+  if (orig_base)                                                        \
+    {                                                                   \
+      if (string1)                                                      \
+        {                                                               \
+          string1_off = string1 - orig_base;                            \
+          end1_off = end1 - orig_base;                                  \
+          end_match_1_off = end_match_1 - orig_base;                    \
+        }                                                               \
+      if (string2)                                                      \
+        {                                                               \
+          string2_off = string2 - orig_base;                            \
+          end2_off = end2 - orig_base;                                  \
+          end_match_2_off = end_match_2 - orig_base;                    \
+        }                                                               \
+      d_off = d - orig_base;                                            \
+      dend_off = dend - orig_base;                                  \
+      dfail_off = dfail - orig_base;                                    \
+    }                                                                   \
   if (!GROW_FAIL_STACK (fail_stack))					\
-    return -2;								\
+    return -2;                                                          \
+  /* GROW_FAIL_STACK may call malloc and relocate the string */         \
+  /* pointers.  */                                                      \
+  re_char* new_base = STR_BASE_PTR (re_match_object);                   \
+  if (new_base && new_base != orig_base)                                \
+    {                                                                   \
+      if (string1)                                                      \
+        {                                                               \
+          string1 = new_base + string1_off;                             \
+          end1 = new_base + end1_off;                                   \
+          end_match_1 = new_base + end_match_1_off;                     \
+        }                                                               \
+      if (string2)                                                      \
+        {                                                               \
+          string2 = new_base + string2_off;                             \
+          end2 = new_base + end2_off;                                   \
+          end_match_2 = new_base + end_match_2_off;                     \
+        }                                                               \
+      d = new_base + d_off;                                             \
+      dend = new_base + dend_off;                                       \
+      dfail = new_base + dfail_off;                                     \
+    }                                                                   \
   DEBUG_PRINT ("\n  Doubled stack; size now: %zd\n", (fail_stack).size);\
   DEBUG_PRINT ("	 slots available: %zd\n", REMAINING_AVAIL_SLOTS);\
 }
@@ -4443,6 +4496,16 @@ re_search_2 (struct re_pattern_buffer *bufp, const char *str1, size_t size1,
 	  && !bufp->can_be_null)
 	return -1;
 
+      /* re_match_2_internal may allocate, causing a relocation of the
+         lisp text object that we're searching.  */
+      ptrdiff_t offset1, offset2;
+      re_char *orig_base = STR_BASE_PTR (re_match_object);
+      if (orig_base)
+        {
+          if (string1) offset1 = string1 - orig_base;
+          if (string2) offset2 = string2 - orig_base;
+        }
+
       val = re_match_2_internal (bufp, string1, size1, string2, size2,
 				 startpos, regs, stop);
 
@@ -4452,6 +4515,13 @@ re_search_2 (struct re_pattern_buffer *bufp, const char *str1, size_t size1,
       if (val == -2)
 	return -2;
 
+      re_char *new_base = STR_BASE_PTR (re_match_object);
+      if (new_base && new_base != orig_base)
+        {
+          if (string1) string1 = offset1 + new_base;
+          if (string2) string2 = offset2 + new_base;
+        }
+
     advance:
       if (!range)
 	break;
@@ -4887,8 +4957,8 @@ WEAK_ALIAS (__re_match, re_match)
 #endif /* not emacs */
 
 #ifdef emacs
-/* In Emacs, this is the string or buffer in which we
-   are matching.  It is used for looking up syntax properties.  */
+/* In Emacs, this is the string or buffer in which we are matching.
+   See the declaration in regex.h for details.  */
 Lisp_Object re_match_object;
 #endif
 
diff --git a/src/regex.h b/src/regex.h
index 51f4424..d5c9690 100644
--- a/src/regex.h
+++ b/src/regex.h
@@ -169,7 +169,8 @@ extern reg_syntax_t re_syntax_options;
 #ifdef emacs
 # include "lisp.h"
 /* In Emacs, this is the string or buffer in which we are matching.
-   It is used for looking up syntax properties.
+   It is used for looking up syntax properties, and also to recompute
+   pointers in case the object is relocated by GC.
 
    If the value is a Lisp string object, we are matching text in that
    string; if it's nil, we are matching text in the current buffer; if
diff --git a/src/search.c b/src/search.c
index dc7e2d8..9a2805d 100644
--- a/src/search.c
+++ b/src/search.c
@@ -287,8 +287,10 @@ looking_at_1 (Lisp_Object string, bool posix)
   immediate_quit = 1;
   QUIT;			/* Do a pending quit right away, to avoid paradoxical behavior */
 
-  /* Get pointers and sizes of the two strings
-     that make up the visible portion of the buffer. */
+  /* Get pointers and sizes of the two strings that make up the
+     visible portion of the buffer.  Note that we can use pointers
+     here, unlike in search_buffer, because we only call re_match_2
+     once.  */
 
   p1 = BEGV_ADDR;
   s1 = GPT_BYTE - BEGV_BYTE;
@@ -407,6 +409,7 @@ string_match_1 (Lisp_Object regexp, Lisp_Object string, Lisp_Object start,
 		   (NILP (Vinhibit_changing_match_data)
 		    ? &search_regs : NULL));
   immediate_quit = 0;
+  re_match_object = Qnil;       /* Stop protecting string from GC.  */
 
   /* Set last_thing_searched only when match data is changed.  */
   if (NILP (Vinhibit_changing_match_data))
@@ -477,6 +480,7 @@ fast_string_match_internal (Lisp_Object regexp, Lisp_Object string,
 		   SBYTES (string), 0,
 		   SBYTES (string), 0);
   immediate_quit = 0;
+  re_match_object = Qnil;       /* Stop protecting string from GC.  */
   return val;
 }
 
@@ -564,6 +568,7 @@ fast_looking_at (Lisp_Object regexp, ptrdiff_t pos, ptrdiff_t pos_byte,
   len = re_match_2 (buf, (char *) p1, s1, (char *) p2, s2,
 		    pos_byte, NULL, limit_byte);
   immediate_quit = 0;
+  re_match_object = Qnil;       /* Stop protecting string from GC.  */
 
   return len;
 }
@@ -1178,8 +1183,8 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 
   if (RE && !(trivial_regexp_p (string) && NILP (Vsearch_spaces_regexp)))
     {
-      unsigned char *p1, *p2;
-      ptrdiff_t s1, s2;
+      unsigned char *base;
+      ptrdiff_t off1, off2, s1, s2;
       struct re_pattern_buffer *bufp;
 
       bufp = compile_pattern (string,
@@ -1193,16 +1198,19 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 				   can take too long. */
       QUIT;			/* Do a pending quit right away,
 				   to avoid paradoxical behavior */
-      /* Get pointers and sizes of the two strings
-	 that make up the visible portion of the buffer. */
+      /* Get offsets and sizes of the two strings that make up the
+         visible portion of the buffer.  We compute offsets instead of
+         pointers because re_search_2 may call malloc and therefore
+         change the buffer text address.  */
 
-      p1 = BEGV_ADDR;
+      base = current_buffer->text->beg;
+      off1 = BEGV_ADDR - base;
       s1 = GPT_BYTE - BEGV_BYTE;
-      p2 = GAP_END_ADDR;
+      off2 = GAP_END_ADDR - base;
       s2 = ZV_BYTE - GPT_BYTE;
       if (s1 < 0)
 	{
-	  p2 = p1;
+          off2 = off1;
 	  s2 = ZV_BYTE - BEGV_BYTE;
 	  s1 = 0;
 	}
@@ -1217,7 +1225,9 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 	{
 	  ptrdiff_t val;
 
-	  val = re_search_2 (bufp, (char *) p1, s1, (char *) p2, s2,
+          val = re_search_2 (bufp,
+                             (char*) (base + off1), s1,
+                             (char*) (base + off2), s2,
 			     pos_byte - BEGV_BYTE, lim_byte - pos_byte,
 			     (NILP (Vinhibit_changing_match_data)
 			      ? &search_regs : &search_regs_1),
@@ -1262,8 +1272,10 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 	{
 	  ptrdiff_t val;
 
-	  val = re_search_2 (bufp, (char *) p1, s1, (char *) p2, s2,
-			     pos_byte - BEGV_BYTE, lim_byte - pos_byte,
+          val = re_search_2 (bufp,
+                             (char*) (base + off1), s1,
+                             (char*) (base + off2), s2,
+                             pos_byte - BEGV_BYTE, lim_byte - pos_byte,
 			     (NILP (Vinhibit_changing_match_data)
 			      ? &search_regs : &search_regs_1),
 			     lim_byte - BEGV_BYTE);
-- 
2.9.3

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: 24358@debbugs.gnu.org,  sam.halliday@gmail.com
> Date: Thu, 20 Oct 2016 00:31:50 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >
> >> +#ifdef emacs
> >> +#define STR_BASE_PTR(obj)                       \
> >> +    (BUFFERP (obj)? XBUFFER (obj)->text->beg :  \
> >> +     STRINGP (obj)? SDATA (obj) :               \
> >> +     NULL)
> >
> 
> [...]
> 
> > the only test in the macro should be STRINGP.
> 
> Hmm, not sure I feel comfortable being that implicit.  I kept this macro
> the same except for using (NILP (obj)? current_buffer->...) instead of
> BUFFERP and XBUFFER.

Fine with me.

> Here is the new patch:

LGTM, a few minor comments below.

> +#ifdef emacs
> +#define STR_BASE_PTR(obj)                   \
> +  (NILP(obj)? current_buffer->text->beg :   \
          ^
Please leave a blank before the left parenthesis where indicated.
Also, another blank between the right parenthesis and the following
question mark.

> +   STRINGP (obj)? SDATA (obj) :             \

Likewise here.

> diff --git a/src/regex.h b/src/regex.h
> index 51f4424..d5c9690 100644
> --- a/src/regex.h
> +++ b/src/regex.h
> @@ -169,7 +169,8 @@ extern reg_syntax_t re_syntax_options;
>  #ifdef emacs
>  # include "lisp.h"
>  /* In Emacs, this is the string or buffer in which we are matching.
> -   It is used for looking up syntax properties.
> +   It is used for looking up syntax properties, and also to recompute
> +   pointers in case the object is relocated by GC.

Not "by GC", but "as a side effect of calling malloc".  Maybe it's a
good idea to also mention ralloc.c here.

> +  /* Get pointers and sizes of the two strings that make up the
> +     visible portion of the buffer.  Note that we can use pointers
> +     here, unlike in search_buffer, because we only call re_match_2
> +     once.  */

I'm not sure the reader will understand the significance of calling
re_match_2 only once.  It would be good to clarify the comment.

Otherwise, I think this should go in.

Thanks!

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

[-- Attachment #1: Type: text/plain, Size: 1368 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:
>
>> +#ifdef emacs
>> +#define STR_BASE_PTR(obj)                   \
>> +  (NILP(obj)? current_buffer->text->beg :   \
>           ^
> Please leave a blank before the left parenthesis where indicated.
> Also, another blank between the right parenthesis and the following
> question mark.
>
>> +   STRINGP (obj)? SDATA (obj) :             \
>
> Likewise here.

Damn, I can't I believe I'm still making these formatting errors,
there's got to be a way to catch these automatically.

>> +   It is used for looking up syntax properties, and also to recompute
>> +   pointers in case the object is relocated by GC.
>
> Not "by GC", but "as a side effect of calling malloc".  Maybe it's a
> good idea to also mention ralloc.c here.
>
>> +  /* Get pointers and sizes of the two strings that make up the
>> +     visible portion of the buffer.  Note that we can use pointers
>> +     here, unlike in search_buffer, because we only call re_match_2
>> +     once.  */
>
> I'm not sure the reader will understand the significance of calling
> re_match_2 only once.  It would be good to clarify the comment.

Okay, I've added some more explanations to these comments.

>
> Otherwise, I think this should go in.

This is for emacs-25, right?  Technically, the bug seems to be present
in 24.5 and earlier, though I can only trigger it in 25.1.


[-- Attachment #2: patch v3 --]
[-- Type: text/plain, Size: 12666 bytes --]

From 97b69a66148c0b28c6d865619b6c1bcee78902a5 Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Wed, 19 Oct 2016 20:23:50 -0400
Subject: [PATCH v3] Fix handling of allocation in regex matching

`re_match_2_internal' uses pointers to the lisp objects that it
searches.  Since it may call malloc when growing the "fail stack", these
pointers may be invalidated while searching, resulting in memory
curruption (Bug #24358).

To fix this, we check the pointer that the lisp object (as specified by
re_match_object) points to before and after growing the stack, and
update existing pointers accordingly.

* src/regex.c (STR_BASE_PTR): New macro.
(ENSURE_FAIL_STACK, re_search_2): Use it to convert pointers into
offsets before possible malloc call, and back into pointers again
afterwards.
(POS_AS_IN_BUFFER): Add explanatory comment about punning trick.
* src/search.c (search_buffer): Instead of storing search location as
pointers, store them as pointers and recompute the corresponding address
for each call to `re_search_2'.
(string_match_1, fast_string_match_internal, fast_looking_at):
* src/dired.c (directory_files_internal): Set `re_match_object' to Qnil
after calling `re_search' or `re_match_2'.
* src/regex.h (re_match_object): Mention new usage in commentary.
---
 src/dired.c  |  4 +++-
 src/regex.c  | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++---
 src/regex.h  |  4 +++-
 src/search.c | 36 ++++++++++++++++++----------
 4 files changed, 103 insertions(+), 17 deletions(-)

diff --git a/src/dired.c b/src/dired.c
index dba575c..006f74c 100644
--- a/src/dired.c
+++ b/src/dired.c
@@ -259,9 +259,11 @@ directory_files_internal (Lisp_Object directory, Lisp_Object full,
       QUIT;
 
       bool wanted = (NILP (match)
-		     || re_search (bufp, SSDATA (name), len, 0, len, 0) >= 0);
+		     || (re_match_object = name,
+                         re_search (bufp, SSDATA (name), len, 0, len, 0) >= 0));
 
       immediate_quit = 0;
+      re_match_object = Qnil;   /* Stop protecting name from GC.  */
 
       if (wanted)
 	{
diff --git a/src/regex.c b/src/regex.c
index 164eb46..1346ef4 100644
--- a/src/regex.c
+++ b/src/regex.c
@@ -152,6 +152,8 @@
 
 /* Converts the pointer to the char to BEG-based offset from the start.  */
 # define PTR_TO_OFFSET(d) POS_AS_IN_BUFFER (POINTER_TO_OFFSET (d))
+/* Strings are 0-indexed, buffers are 1-indexed; we pun on the boolean
+   result to get the right base index.  */
 # define POS_AS_IN_BUFFER(p) ((p) + (NILP (re_match_object) || BUFFERP (re_match_object)))
 
 # define RE_MULTIBYTE_P(bufp) ((bufp)->multibyte)
@@ -1436,11 +1438,62 @@ WEAK_ALIAS (__re_set_syntax, re_set_syntax)
 #define NEXT_FAILURE_HANDLE(h) fail_stack.stack[(h) - 3].integer
 #define TOP_FAILURE_HANDLE() fail_stack.frame
 
+#ifdef emacs
+#define STR_BASE_PTR(obj)                     \
+  (NILP (obj) ? current_buffer->text->beg :   \
+   STRINGP (obj) ? SDATA (obj) :              \
+   NULL)
+#else
+#define STR_BASE_PTR(obj) NULL
+#endif
 
 #define ENSURE_FAIL_STACK(space)					\
 while (REMAINING_AVAIL_SLOTS <= space) {				\
+  re_char* orig_base = STR_BASE_PTR (re_match_object);                  \
+  ptrdiff_t string1_off, end1_off, end_match_1_off;                     \
+  ptrdiff_t string2_off, end2_off, end_match_2_off;                     \
+  ptrdiff_t d_off, dend_off, dfail_off;                                 \
+  if (orig_base)                                                        \
+    {                                                                   \
+      if (string1)                                                      \
+        {                                                               \
+          string1_off = string1 - orig_base;                            \
+          end1_off = end1 - orig_base;                                  \
+          end_match_1_off = end_match_1 - orig_base;                    \
+        }                                                               \
+      if (string2)                                                      \
+        {                                                               \
+          string2_off = string2 - orig_base;                            \
+          end2_off = end2 - orig_base;                                  \
+          end_match_2_off = end_match_2 - orig_base;                    \
+        }                                                               \
+      d_off = d - orig_base;                                            \
+      dend_off = dend - orig_base;                                      \
+      dfail_off = dfail - orig_base;                                    \
+    }                                                                   \
   if (!GROW_FAIL_STACK (fail_stack))					\
-    return -2;								\
+    return -2;                                                          \
+  /* GROW_FAIL_STACK may call malloc and relocate the string */         \
+  /* pointers.  */                                                      \
+  re_char* new_base = STR_BASE_PTR (re_match_object);                   \
+  if (new_base && new_base != orig_base)                                \
+    {                                                                   \
+      if (string1)                                                      \
+        {                                                               \
+          string1 = new_base + string1_off;                             \
+          end1 = new_base + end1_off;                                   \
+          end_match_1 = new_base + end_match_1_off;                     \
+        }                                                               \
+      if (string2)                                                      \
+        {                                                               \
+          string2 = new_base + string2_off;                             \
+          end2 = new_base + end2_off;                                   \
+          end_match_2 = new_base + end_match_2_off;                     \
+        }                                                               \
+      d = new_base + d_off;                                             \
+      dend = new_base + dend_off;                                       \
+      dfail = new_base + dfail_off;                                     \
+    }                                                                   \
   DEBUG_PRINT ("\n  Doubled stack; size now: %zd\n", (fail_stack).size);\
   DEBUG_PRINT ("	 slots available: %zd\n", REMAINING_AVAIL_SLOTS);\
 }
@@ -4443,6 +4496,16 @@ re_search_2 (struct re_pattern_buffer *bufp, const char *str1, size_t size1,
 	  && !bufp->can_be_null)
 	return -1;
 
+      /* re_match_2_internal may allocate, causing a relocation of the
+         lisp text object that we're searching.  */
+      ptrdiff_t offset1, offset2;
+      re_char *orig_base = STR_BASE_PTR (re_match_object);
+      if (orig_base)
+        {
+          if (string1) offset1 = string1 - orig_base;
+          if (string2) offset2 = string2 - orig_base;
+        }
+
       val = re_match_2_internal (bufp, string1, size1, string2, size2,
 				 startpos, regs, stop);
 
@@ -4452,6 +4515,13 @@ re_search_2 (struct re_pattern_buffer *bufp, const char *str1, size_t size1,
       if (val == -2)
 	return -2;
 
+      re_char *new_base = STR_BASE_PTR (re_match_object);
+      if (new_base && new_base != orig_base)
+        {
+          if (string1) string1 = offset1 + new_base;
+          if (string2) string2 = offset2 + new_base;
+        }
+
     advance:
       if (!range)
 	break;
@@ -4887,8 +4957,8 @@ WEAK_ALIAS (__re_match, re_match)
 #endif /* not emacs */
 
 #ifdef emacs
-/* In Emacs, this is the string or buffer in which we
-   are matching.  It is used for looking up syntax properties.  */
+/* In Emacs, this is the string or buffer in which we are matching.
+   See the declaration in regex.h for details.  */
 Lisp_Object re_match_object;
 #endif
 
diff --git a/src/regex.h b/src/regex.h
index 51f4424..61c771c 100644
--- a/src/regex.h
+++ b/src/regex.h
@@ -169,7 +169,9 @@ extern reg_syntax_t re_syntax_options;
 #ifdef emacs
 # include "lisp.h"
 /* In Emacs, this is the string or buffer in which we are matching.
-   It is used for looking up syntax properties.
+   It is used for looking up syntax properties, and also to recompute
+   pointers in case the object is relocated as a side effect of
+   calling malloc (if it calls r_alloc_sbrk in ralloc.c).
 
    If the value is a Lisp string object, we are matching text in that
    string; if it's nil, we are matching text in the current buffer; if
diff --git a/src/search.c b/src/search.c
index dc7e2d8..ec5a1d7 100644
--- a/src/search.c
+++ b/src/search.c
@@ -287,8 +287,10 @@ looking_at_1 (Lisp_Object string, bool posix)
   immediate_quit = 1;
   QUIT;			/* Do a pending quit right away, to avoid paradoxical behavior */
 
-  /* Get pointers and sizes of the two strings
-     that make up the visible portion of the buffer. */
+  /* Get pointers and sizes of the two strings that make up the
+     visible portion of the buffer.  Note that we can use pointers
+     here, unlike in search_buffer, because we only call re_match_2
+     once, after which we never use the pointers again.  */
 
   p1 = BEGV_ADDR;
   s1 = GPT_BYTE - BEGV_BYTE;
@@ -407,6 +409,7 @@ string_match_1 (Lisp_Object regexp, Lisp_Object string, Lisp_Object start,
 		   (NILP (Vinhibit_changing_match_data)
 		    ? &search_regs : NULL));
   immediate_quit = 0;
+  re_match_object = Qnil;       /* Stop protecting string from GC.  */
 
   /* Set last_thing_searched only when match data is changed.  */
   if (NILP (Vinhibit_changing_match_data))
@@ -477,6 +480,7 @@ fast_string_match_internal (Lisp_Object regexp, Lisp_Object string,
 		   SBYTES (string), 0,
 		   SBYTES (string), 0);
   immediate_quit = 0;
+  re_match_object = Qnil;       /* Stop protecting string from GC.  */
   return val;
 }
 
@@ -564,6 +568,7 @@ fast_looking_at (Lisp_Object regexp, ptrdiff_t pos, ptrdiff_t pos_byte,
   len = re_match_2 (buf, (char *) p1, s1, (char *) p2, s2,
 		    pos_byte, NULL, limit_byte);
   immediate_quit = 0;
+  re_match_object = Qnil;       /* Stop protecting string from GC.  */
 
   return len;
 }
@@ -1178,8 +1183,8 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 
   if (RE && !(trivial_regexp_p (string) && NILP (Vsearch_spaces_regexp)))
     {
-      unsigned char *p1, *p2;
-      ptrdiff_t s1, s2;
+      unsigned char *base;
+      ptrdiff_t off1, off2, s1, s2;
       struct re_pattern_buffer *bufp;
 
       bufp = compile_pattern (string,
@@ -1193,16 +1198,19 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 				   can take too long. */
       QUIT;			/* Do a pending quit right away,
 				   to avoid paradoxical behavior */
-      /* Get pointers and sizes of the two strings
-	 that make up the visible portion of the buffer. */
+      /* Get offsets and sizes of the two strings that make up the
+         visible portion of the buffer.  We compute offsets instead of
+         pointers because re_search_2 may call malloc and therefore
+         change the buffer text address.  */
 
-      p1 = BEGV_ADDR;
+      base = current_buffer->text->beg;
+      off1 = BEGV_ADDR - base;
       s1 = GPT_BYTE - BEGV_BYTE;
-      p2 = GAP_END_ADDR;
+      off2 = GAP_END_ADDR - base;
       s2 = ZV_BYTE - GPT_BYTE;
       if (s1 < 0)
 	{
-	  p2 = p1;
+          off2 = off1;
 	  s2 = ZV_BYTE - BEGV_BYTE;
 	  s1 = 0;
 	}
@@ -1217,7 +1225,9 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 	{
 	  ptrdiff_t val;
 
-	  val = re_search_2 (bufp, (char *) p1, s1, (char *) p2, s2,
+          val = re_search_2 (bufp,
+                             (char*) (base + off1), s1,
+                             (char*) (base + off2), s2,
 			     pos_byte - BEGV_BYTE, lim_byte - pos_byte,
 			     (NILP (Vinhibit_changing_match_data)
 			      ? &search_regs : &search_regs_1),
@@ -1262,8 +1272,10 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 	{
 	  ptrdiff_t val;
 
-	  val = re_search_2 (bufp, (char *) p1, s1, (char *) p2, s2,
-			     pos_byte - BEGV_BYTE, lim_byte - pos_byte,
+          val = re_search_2 (bufp,
+                             (char*) (base + off1), s1,
+                             (char*) (base + off2), s2,
+                             pos_byte - BEGV_BYTE, lim_byte - pos_byte,
 			     (NILP (Vinhibit_changing_match_data)
 			      ? &search_regs : &search_regs_1),
 			     lim_byte - BEGV_BYTE);
-- 
2.9.3

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: 24358@debbugs.gnu.org,  sam.halliday@gmail.com
> Date: Thu, 20 Oct 2016 21:22:25 -0400
> 
> > Otherwise, I think this should go in.
> 
> This is for emacs-25, right?  Technically, the bug seems to be present
> in 24.5 and earlier, though I can only trigger it in 25.1.

Yes, please push to emacs-25.  The bug is sufficiently nasty and the
solution simple enough to justify that.

Thanks.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

# This one was about the memory corruption,
# for the max-specpdl-size problem, see Bug #24751.
retitle 24358 Memory corruption after r_alloc_sbrk during regex matching
found 24358 25.1
tags 24358 fixed
close 24358 25.2
quit

Eli Zaretskii <eliz@gnu.org> writes:

>> From: npostavs@users.sourceforge.net
>> Cc: 24358@debbugs.gnu.org,  sam.halliday@gmail.com
>> Date: Thu, 20 Oct 2016 21:22:25 -0400
>> 
>> > Otherwise, I think this should go in.
>> 
>> This is for emacs-25, right?  Technically, the bug seems to be present
>> in 24.5 and earlier, though I can only trigger it in 25.1.
>
> Yes, please push to emacs-25.  The bug is sufficiently nasty and the
> solution simple enough to justify that.

I've pushed as ad66b3fa.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Sam Halliday]

[-- Attachment #1: Type: text/plain, Size: 932 bytes --]

when I build emacs-25 with this patch (either by building emacs-25
directly or taking emacs-25.1 and applying this patch) I get a failure
in make:

./autogen.sh
CFLAGS='-O0 -g3 -gdwarf-4' ./configure --enable-checking='yes,glyphs' --enable-check-lisp-object-type --prefix=/opt/emacs25
make

.
.
.
make[2]: Leaving directory '/home/fommil/build/emacs-git/emacs/admin/grammars'
collected 50% ...
collected 60% ...
collected 70% ...
collected 80% ...
collected 90% ...
Processing OKURI-NASI entries ...
processed 10% ...
processed 20% ...
Args out of range: #("崎/
かん" 0 5 (charset japanese-jisx0208)), -6, nil
make[2]: *** [Makefile:142: ../lisp/leim/ja-dic/ja-dic.el] Error 255
make[2]: Leaving directory '/home/fommil/build/emacs-git/emacs/leim'
make[1]: *** [Makefile:320: leim] Error 2
make[1]: Leaving directory '/home/fommil/build/emacs-git/emacs/lisp'
make: *** [Makefile:385: lisp] Error 2



[-- Attachment #2.1: Type: text/plain, Size: 851 bytes --]

npostavs@users.sourceforge.net writes:

> # This one was about the memory corruption,
> # for the max-specpdl-size problem, see Bug #24751.
> retitle 24358 Memory corruption after r_alloc_sbrk during regex matching
> found 24358 25.1
> tags 24358 fixed
> close 24358 25.2
> quit
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: npostavs@users.sourceforge.net
>>> Cc: 24358@debbugs.gnu.org,  sam.halliday@gmail.com
>>> Date: Thu, 20 Oct 2016 21:22:25 -0400
>>> 
>>> > Otherwise, I think this should go in.
>>> 
>>> This is for emacs-25, right?  Technically, the bug seems to be present
>>> in 24.5 and earlier, though I can only trigger it in 25.1.
>>
>> Yes, please push to emacs-25.  The bug is sufficiently nasty and the
>> solution simple enough to justify that.
>
> I've pushed as ad66b3fa.

-- 
Best regards,
Sam

[-- Attachment #2.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 162 bytes --]

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

Sam Halliday <sam.halliday@gmail.com> writes:

> when I build emacs-25 with this patch (either by building emacs-25
> directly or taking emacs-25.1 and applying this patch) I get a failure
> in make:
>
> ./autogen.sh
> CFLAGS='-O0 -g3 -gdwarf-4' ./configure --enable-checking='yes,glyphs' --enable-check-lisp-object-type --prefix=/opt/emacs25
> make
>
> .
> .
> .
> make[2]: Leaving directory '/home/fommil/build/emacs-git/emacs/admin/grammars'
> collected 50% ...
> collected 60% ...
> collected 70% ...
> collected 80% ...
> collected 90% ...
> Processing OKURI-NASI entries ...
> processed 10% ...
> processed 20% ...
> Args out of range: #("崎/
> かん" 0 5 (charset japanese-jisx0208)), -6, nil
> make[2]: *** [Makefile:142: ../lisp/leim/ja-dic/ja-dic.el] Error 255
> make[2]: Leaving directory '/home/fommil/build/emacs-git/emacs/leim'
> make[1]: *** [Makefile:320: leim] Error 2
> make[1]: Leaving directory '/home/fommil/build/emacs-git/emacs/lisp'
> make: *** [Makefile:385: lisp] Error 2
>

I can't reproduce this.  I did a 'make extraclean' in lisp/leim, and
then ran 'make', ja-dic.el was generated without error:

  GEN      ../lisp/leim/ja-dic/ja-dic.el
Reading file "/home/npostavs/src/emacs/emacs-25/leim/SKK-DIC/SKK-JISYO.L" ...
Processing OKURI-ARI entries ...
Processing POSTFIX entries ...
Processing PREFIX entries ...
Collecting OKURI-NASI entries ...
collected 26% ...
collected 30% ...
collected 40% ...
collected 50% ...
collected 60% ...
collected 70% ...
collected 80% ...
collected 90% ...
Processing OKURI-NASI entries ...
processed 10% ...
processed 20% ...
processed 30% ...
processed 40% ...
processed 50% ...
processed 60% ...
processed 70% ...
processed 80% ...
processed 90% ...
processed 100% ...
make[2]: Leaving directory '/home/npostavs/src/emacs/emacs-25/leim'

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: Eli Zaretskii <eliz@gnu.org>,  24358@debbugs.gnu.org
> Date: Sat, 22 Oct 2016 18:46:43 -0400
> 
> Sam Halliday <sam.halliday@gmail.com> writes:
> 
> > Processing OKURI-NASI entries ...
> > processed 10% ...
> > processed 20% ...
> > Args out of range: #("崎/
> > かん" 0 5 (charset japanese-jisx0208)), -6, nil
> > make[2]: *** [Makefile:142: ../lisp/leim/ja-dic/ja-dic.el] Error 255
> > make[2]: Leaving directory '/home/fommil/build/emacs-git/emacs/leim'
> > make[1]: *** [Makefile:320: leim] Error 2
> > make[1]: Leaving directory '/home/fommil/build/emacs-git/emacs/lisp'
> > make: *** [Makefile:385: lisp] Error 2
> >
> 
> I can't reproduce this.

Neither can I.

Sam, does your build use ralloc.c?  Also, are you sure this change
causes the problem?  Note that Paul pushed a follow-up change after
that: do you have it in your tree?

> I did a 'make extraclean' in lisp/leim, and
> then ran 'make', ja-dic.el was generated without error:

Sam, please run the failed command under GDB, put a breakpoint in
args_out_of_range, and show the C and Lisp backtrace when this error
happens.  (If args_out_of_range is called more than once, then please
make sure you are showing the correct instance, e.g. by looking at the
"processed NN%" lines.)

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Sam Halliday]

I don't know which alloc my build is using, I'm on archlinux and I
expect it will use the same impl that was causing the problems earlier
in the thread.

I also got the problem on the tip of emacs-25, but it's hard for me to
see which commits are relevant to this ticket because they've all been
squashed into the mainline instead of merged in.

I'm afraid I'm travelling today and cannot run these commands, I will
try to do this shortly.  How do I work out the command that make is
running?


On 23 October 2016 at 07:41, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: npostavs@users.sourceforge.net
>> Cc: Eli Zaretskii <eliz@gnu.org>,  24358@debbugs.gnu.org
>> Date: Sat, 22 Oct 2016 18:46:43 -0400
>>
>> Sam Halliday <sam.halliday@gmail.com> writes:
>>
>> > Processing OKURI-NASI entries ...
>> > processed 10% ...
>> > processed 20% ...
>> > Args out of range: #("崎/
>> > かん" 0 5 (charset japanese-jisx0208)), -6, nil
>> > make[2]: *** [Makefile:142: ../lisp/leim/ja-dic/ja-dic.el] Error 255
>> > make[2]: Leaving directory '/home/fommil/build/emacs-git/emacs/leim'
>> > make[1]: *** [Makefile:320: leim] Error 2
>> > make[1]: Leaving directory '/home/fommil/build/emacs-git/emacs/lisp'
>> > make: *** [Makefile:385: lisp] Error 2
>> >
>>
>> I can't reproduce this.
>
> Neither can I.
>
> Sam, does your build use ralloc.c?  Also, are you sure this change
> causes the problem?  Note that Paul pushed a follow-up change after
> that: do you have it in your tree?
>
>> I did a 'make extraclean' in lisp/leim, and
>> then ran 'make', ja-dic.el was generated without error:
>
> Sam, please run the failed command under GDB, put a breakpoint in
> args_out_of_range, and show the C and Lisp backtrace when this error
> happens.  (If args_out_of_range is called more than once, then please
> make sure you are showing the correct instance, e.g. by looking at the
> "processed NN%" lines.)

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: Sam Halliday <sam.halliday@gmail.com>
> Date: Sun, 23 Oct 2016 09:57:21 +0100
> Cc: npostavs@users.sourceforge.net, 24358@debbugs.gnu.org
> 
> I don't know which alloc my build is using, I'm on archlinux and I
> expect it will use the same impl that was causing the problems earlier
> in the thread.

Do you have ralloc.o in src/?  If you do, you are using ralloc.c.

> I'm afraid I'm travelling today and cannot run these commands, I will
> try to do this shortly.  How do I work out the command that make is
> running?

Like this:

  cd leim
  make V=1

This will show the command which creates ja-dic/ja-dic.el.  (You can
also see it in leim/Makefile.)

Thanks.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Sam Halliday]

ralloc.c is in "src".

Running the failing command, the breakpoint didn't take... must be
something else.


EMACSLOADPATH= gdb ../src/emacs
GNU gdb (GDB) 7.12
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ../src/emacs...done.
(gdb)  break args_out_of_range
Breakpoint 1 at 0x614cc1: file data.c, line 163.
(gdb) run -batch --no-site-file --no-site-lisp -batch -l ja-dic-cnv -f
batch-skkdic-convert -dir "./../lisp/leim/ja-dic"
"SKK-DIC/SKK-JISYO.L"
Starting program: /home/fommil/build/emacs-git/emacs/src/emacs -batch
--no-site-file --no-site-lisp -batch -l ja-dic-cnv -f
batch-skkdic-convert -dir "./../lisp/leim/ja-dic"
"SKK-DIC/SKK-JISYO.L"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
[New Thread 0x7fffe72e5700 (LWP 14444)]
Reading file "/home/fommil/build/emacs-git/emacs/leim/SKK-DIC/SKK-JISYO.L" ...
Processing OKURI-ARI entries ...
Processing POSTFIX entries ...
Processing PREFIX entries ...
Collecting OKURI-NASI entries ...
collected 26% ...
collected 30% ...
collected 40% ...
collected 50% ...
collected 60% ...
collected 70% ...
collected 80% ...
collected 90% ...
Processing OKURI-NASI entries ...
processed 10% ...
processed 20% ...
processed 30% ...
processed 40% ...
Args out of range: #("七宝焼/" 0 4 (charset japanese-jisx0208)), -5, nil
[Thread 0x7ffff7f246c0 (LWP 14440) exited]
[Inferior 1 (process 14440) exited with code 0377]
(gdb)


On 23 October 2016 at 10:19, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Sam Halliday <sam.halliday@gmail.com>
>> Date: Sun, 23 Oct 2016 09:57:21 +0100
>> Cc: npostavs@users.sourceforge.net, 24358@debbugs.gnu.org
>>
>> I don't know which alloc my build is using, I'm on archlinux and I
>> expect it will use the same impl that was causing the problems earlier
>> in the thread.
>
> Do you have ralloc.o in src/?  If you do, you are using ralloc.c.
>
>> I'm afraid I'm travelling today and cannot run these commands, I will
>> try to do this shortly.  How do I work out the command that make is
>> running?
>
> Like this:
>
>   cd leim
>   make V=1
>
> This will show the command which creates ja-dic/ja-dic.el.  (You can
> also see it in leim/Makefile.)
>
> Thanks.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: Sam Halliday <sam.halliday@gmail.com>
> Date: Sun, 23 Oct 2016 14:40:26 +0100
> Cc: npostavs@users.sourceforge.net, 24358@debbugs.gnu.org
> 
> ralloc.c is in "src".

What about ralloc.o?

> Running the failing command, the breakpoint didn't take... must be
> something else.

Probably some code calls 'signal' or 'error' directly.  A breakpoint
in Fsignal should catch that.  Unfortunately, it could trigger many
times before, so if that happens, let me know if you need help in
making the breakpoint more selective.

> Processing OKURI-NASI entries ...
> processed 10% ...
> processed 20% ...
> processed 30% ...
> processed 40% ...
> Args out of range: #("七宝焼/" 0 4 (charset japanese-jisx0208)), -5, nil

Note that the error is slightly different now (a different string and
the argument is -5 instead of -6), and also in a slightly different
place of the build (after 40% instead of 20%).

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Sam Halliday]

On 23 October 2016 at 15:07, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Sam Halliday <sam.halliday@gmail.com>
>>
>> ralloc.c is in "src".
>
> What about ralloc.o?

yes ralloc.o is there, and of course this is what I should have looked
for before because ralloc.c is clearly version controlled, sorry! :-)

>> Running the failing command, the breakpoint didn't take... must be
>> something else.
>
> Probably some code calls 'signal' or 'error' directly.  A breakpoint
> in Fsignal should catch that.  Unfortunately, it could trigger many
> times before, so if that happens, let me know if you need help in
> making the breakpoint more selective.
>
>> Processing OKURI-NASI entries ...
>> processed 10% ...
>> processed 20% ...
>> processed 30% ...
>> processed 40% ...
>> Args out of range: #("七宝焼/" 0 4 (charset japanese-jisx0208)), -5, nil
>
> Note that the error is slightly different now (a different string and
> the argument is -5 instead of -6), and also in a slightly different
> place of the build (after 40% instead of 20%).

breaking on Fsignal is hitting the problem... not sure what to do
here. I can type "bt" but that's where my gdb skillz end... do you
want me to do anything else or is this enough? I'd imagine seeing the
contents of error_symbol and data would be pretty useful. I'm not sure
the etc/DEBUG .gdbinit file is being loaded properly when invoked from
the leim directory.

EMACSLOADPATH= gdb ../src/emacs
GNU gdb (GDB) 7.12
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ../src/emacs...done.
(gdb) break Fsignal
Breakpoint 1 at 0x634e00: file eval.c, line 1476.
(gdb) run -batch --no-site-file --no-site-lisp -batch -l ja-dic-cnv -f
batch-skkdic-convert -dir "./../lisp/leim/ja-dic"
"SKK-DIC/SKK-JISYO.L"
Starting program: /home/fommil/build/emacs-git/emacs/src/emacs -batch
--no-site-file --no-site-lisp -batch -l ja-dic-cnv -f
batch-skkdic-convert -dir "./../lisp/leim/ja-dic"
"SKK-DIC/SKK-JISYO.L"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
[New Thread 0x7fffe72e5700 (LWP 21567)]
Reading file "/home/fommil/build/emacs-git/emacs/leim/SKK-DIC/SKK-JISYO.L" ...
Processing OKURI-ARI entries ...
Processing POSTFIX entries ...
Processing PREFIX entries ...
Collecting OKURI-NASI entries ...
collected 26% ...
collected 30% ...
collected 40% ...
collected 50% ...
collected 60% ...
collected 70% ...
collected 80% ...
collected 90% ...
Processing OKURI-NASI entries ...
processed 10% ...
processed 20% ...
processed 30% ...
processed 40% ...

Thread 1 "emacs" hit Breakpoint 1, Fsignal (error_symbol=...,
data=...) at eval.c:1476
1476    = (NILP (error_symbol) ? Fcar (data) : error_symbol);
(gdb) bt
#0  0x0000000000634e00 in Fsignal (error_symbol=..., data=...) at eval.c:1476
#1  0x00000000006351d9 in xsignal (error_symbol=..., data=...) at eval.c:1582
#2  0x00000000006352ae in xsignal3 (error_symbol=..., arg1=...,
arg2=..., arg3=...) at eval.c:1609
#3  0x0000000000614d16 in args_out_of_range_3 (a1=..., a2=..., a3=...)
at data.c:169
#4  0x000000000063f687 in validate_subarray (array=..., from=...,
to=..., size=4, ifrom=0x7fffffff6e18, ito=0x7fffffff6e10) at
fns.c:1210
#5  0x000000000063f70c in Fsubstring (string=..., from=..., to=...) at
fns.c:1235
#6  0x0000000000636da4 in eval_sub (form=...) at eval.c:2177
#7  0x0000000000636c98 in eval_sub (form=...) at eval.c:2159
#8  0x0000000000631bb4 in Fif (args=...) at eval.c:385
#9  0x00000000006369df in eval_sub (form=...) at eval.c:2124
#10 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#11 0x0000000000633be9 in Fwhile (args=...) at eval.c:970
#12 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#13 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#14 0x0000000000633fb0 in internal_catch (tag=..., func=0x631e2e
<Fprogn>, arg=...) at eval.c:1079
#15 0x0000000000633f64 in Fcatch (args=...) at eval.c:1056
#16 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#17 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#18 0x0000000000633ad3 in Flet (args=...) at eval.c:951
#19 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#20 0x0000000000631afc in Fand (args=...) at eval.c:366
#21 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#22 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#23 0x0000000000633ad3 in Flet (args=...) at eval.c:951
#24 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#25 0x0000000000631afc in Fand (args=...) at eval.c:366
#26 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#27 0x0000000000631a31 in For (args=...) at eval.c:346
#28 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#29 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#30 0x0000000000633ad3 in Flet (args=...) at eval.c:951
#31 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#32 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#33 0x0000000000639184 in funcall_lambda (fun=..., nargs=5,
arg_vector=0x0) at eval.c:2921
#34 0x00000000006389cd in apply_lambda (fun=..., args=..., count=77)
at eval.c:2799
#35 0x0000000000637145 in eval_sub (form=...) at eval.c:2246
#36 0x0000000000631a31 in For (args=...) at eval.c:346
#37 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#38 0x0000000000631afc in Fand (args=...) at eval.c:366
#39 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#40 0x0000000000631a31 in For (args=...) at eval.c:346
#41 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#42 0x0000000000631bb4 in Fif (args=...) at eval.c:385
#43 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#44 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#45 0x0000000000633ad3 in Flet (args=...) at eval.c:951
#46 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#47 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#48 0x0000000000633be9 in Fwhile (args=...) at eval.c:970
#49 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#50 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#51 0x0000000000633ad3 in Flet (args=...) at eval.c:951
#52 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#53 0x0000000000631afc in Fand (args=...) at eval.c:366
#54 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#55 0x0000000000631a31 in For (args=...) at eval.c:346
#56 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#57 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#58 0x0000000000633ad3 in Flet (args=...) at eval.c:951
#59 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#60 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#61 0x0000000000639184 in funcall_lambda (fun=..., nargs=6,
arg_vector=0x0) at eval.c:2921
#62 0x00000000006389cd in apply_lambda (fun=..., args=..., count=55)
at eval.c:2799
#63 0x0000000000637145 in eval_sub (form=...) at eval.c:2246
#64 0x0000000000636c98 in eval_sub (form=...) at eval.c:2159
#65 0x0000000000631afc in Fand (args=...) at eval.c:366
#66 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#67 0x0000000000631a31 in For (args=...) at eval.c:346
#68 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#69 0x0000000000631bb4 in Fif (args=...) at eval.c:385
#70 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#71 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#72 0x0000000000633be9 in Fwhile (args=...) at eval.c:970
#73 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#74 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#75 0x0000000000633ad3 in Flet (args=...) at eval.c:951
#76 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#77 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#78 0x0000000000639184 in funcall_lambda (fun=..., nargs=3,
arg_vector=0x0) at eval.c:2921
#79 0x00000000006389cd in apply_lambda (fun=..., args=..., count=43)
at eval.c:2799
#80 0x0000000000637145 in eval_sub (form=...) at eval.c:2246
#81 0x0000000000632122 in Fsetq (args=...) at eval.c:502
#82 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#83 0x0000000000631bb4 in Fif (args=...) at eval.c:385
#84 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#85 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#86 0x0000000000633ad3 in Flet (args=...) at eval.c:951
#87 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#88 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#89 0x0000000000633be9 in Fwhile (args=...) at eval.c:970
#90 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#91 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#92 0x0000000000633ad3 in Flet (args=...) at eval.c:951
#93 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#94 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#95 0x0000000000622ec9 in Fsave_current_buffer (args=...) at editfns.c:1027
#96 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#97 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#98 0x0000000000639184 in funcall_lambda (fun=..., nargs=2,
arg_vector=0x0) at eval.c:2921
#99 0x00000000006389cd in apply_lambda (fun=..., args=..., count=27)
at eval.c:2799
#100 0x0000000000637145 in eval_sub (form=...) at eval.c:2246
#101 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#102 0x0000000000633ad3 in Flet (args=...) at eval.c:951
#103 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#104 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#105 0x0000000000622ec9 in Fsave_current_buffer (args=...) at editfns.c:1027
#106 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#107 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#108 0x0000000000633566 in FletX (args=...) at eval.c:887
#109 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#110 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#111 0x0000000000639184 in funcall_lambda (fun=..., nargs=2,
arg_vector=0x0) at eval.c:2921
#112 0x00000000006389cd in apply_lambda (fun=..., args=..., count=15)
at eval.c:2799
#113 0x0000000000637145 in eval_sub (form=...) at eval.c:2246
#114 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#115 0x0000000000633ad3 in Flet (args=...) at eval.c:951
#116 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#117 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#118 0x0000000000631cbc in Fif (args=...) at eval.c:389
#119 0x00000000006369df in eval_sub (form=...) at eval.c:2124
#120 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
#121 0x0000000000639184 in funcall_lambda (fun=..., nargs=0,
arg_vector=0x0) at eval.c:2921
#122 0x000000000063875e in Ffuncall (nargs=1, args=0x7fffffffc6e0) at
eval.c:2759
#123 0x0000000000683e8a in exec_byte_code (bytestr=..., vector=...,
maxdepth=..., args_template=..., nargs=1, args=0x7fffffffcf88) at
bytecode.c:880
#124 0x0000000000638d80 in funcall_lambda (fun=..., nargs=1,
arg_vector=0x7fffffffcf80) at eval.c:2862
#125 0x0000000000638659 in Ffuncall (nargs=2, args=0x7fffffffcf78) at
eval.c:2747
#126 0x0000000000683e8a in exec_byte_code (bytestr=..., vector=...,
maxdepth=..., args_template=..., nargs=0, args=0x7fffffffd878) at
bytecode.c:880
#127 0x0000000000638d80 in funcall_lambda (fun=..., nargs=0,
arg_vector=0x7fffffffd878) at eval.c:2862
#128 0x0000000000638659 in Ffuncall (nargs=1, args=0x7fffffffd870) at
eval.c:2747
#129 0x0000000000683e8a in exec_byte_code (bytestr=..., vector=...,
maxdepth=..., args_template=..., nargs=0, args=0x7fffffffe040) at
bytecode.c:880
#130 0x0000000000638d80 in funcall_lambda (fun=..., nargs=0,
arg_vector=0x7fffffffe040) at eval.c:2862
#131 0x00000000006389cd in apply_lambda (fun=..., args=..., count=4)
at eval.c:2799
#132 0x0000000000636f3e in eval_sub (form=...) at eval.c:2216
#133 0x00000000006363f8 in Feval (form=..., lexical=...) at eval.c:1993
#134 0x0000000000587ce2 in top_level_2 () at keyboard.c:1121
#135 0x00000000006349f6 in internal_condition_case (bfun=0x587cc5
<top_level_2>, handlers=..., hfun=0x5876f3 <cmd_error>) at eval.c:1314
#136 0x0000000000587d23 in top_level_1 (ignore=...) at keyboard.c:1129
#137 0x0000000000633fb0 in internal_catch (tag=..., func=0x587ce4
<top_level_1>, arg=...) at eval.c:1079
#138 0x0000000000587c1d in command_loop () at keyboard.c:1090
#139 0x00000000005871f5 in recursive_edit_1 () at keyboard.c:697
#140 0x00000000005873ec in Frecursive_edit () at keyboard.c:768
#141 0x0000000000585152 in main (argc=11, argv=0x7fffffffe5d8) at emacs.c:1626

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: Sam Halliday <sam.halliday@gmail.com>
> Date: Sun, 23 Oct 2016 16:42:20 +0100
> Cc: npostavs@users.sourceforge.net, 24358@debbugs.gnu.org
> 
> breaking on Fsignal is hitting the problem... not sure what to do
> here. I can type "bt" but that's where my gdb skillz end... do you
> want me to do anything else or is this enough?

"bt" is a good start, thanks.  After that, the results of "xbacktrace"
will help more.

> I'd imagine seeing the contents of error_symbol and data would be
> pretty useful. I'm not sure the etc/DEBUG .gdbinit file is being
> loaded properly when invoked from the leim directory.

You can read .gdbinit manually:

  (gdb) source ../src/.gdbinit

And yes, making sure error_symbol and data are as we expect them would
be a good start.

Thanks.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Sam Halliday]

ok, that worked to source the .gdbinit and xbacktrace works now...
(this is a fresh session).

I don't know how to print out the parameters to this function. I tried
"info" (see below) but it's still giving ...


EMACSLOADPATH= gdb ../src/emacs
GNU gdb (GDB) 7.12
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ../src/emacs...done.
(gdb) break Fsigna
Function "Fsigna" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (Fsigna) pending.
(gdb) break Fsignal
Breakpoint 2 at 0x634e00: file eval.c, line 1476.
(gdb) source ../src/.gdbinit
SIGINT is used by the debugger.
Are you sure you want to change it? (y or n) [answered Y; input not
from terminal]
DISPLAY = :0
TERM = rxvt-unicode-256color
Breakpoint 3 at 0x583686: file emacs.c, line 354.
Temporary breakpoint 4 at 0x5ae0bc: file sysdep.c, line 915.
(gdb) run -batch --no-site-file --no-site-lisp -batch -l ja-dic-cnv -f
batch-skkdic-convert -dir "./../lisp/leim/ja-dic"
"SKK-DIC/SKK-JISYO.L"
Starting program: /home/fommil/build/emacs-git/emacs/src/emacs -batch
--no-site-file --no-site-lisp -batch -l ja-dic-cnv -f
batch-skkdic-convert -dir "./../lisp/leim/ja-dic"
"SKK-DIC/SKK-JISYO.L"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
[New Thread 0x7fffe72e5700 (LWP 24197)]
Reading file "/home/fommil/build/emacs-git/emacs/leim/SKK-DIC/SKK-JISYO.L" ...
Processing OKURI-ARI entries ...
Processing POSTFIX entries ...
Processing PREFIX entries ...
Collecting OKURI-NASI entries ...
collected 26% ...
collected 30% ...
collected 40% ...
collected 50% ...
collected 60% ...
collected 70% ...
collected 80% ...
collected 90% ...
Processing OKURI-NASI entries ...
processed 10% ...
processed 20% ...
processed 30% ...
processed 40% ...

Thread 1 "emacs" hit Breakpoint 2, Fsignal (error_symbol=...,
data=...) at eval.c:1476
1476    = (NILP (error_symbol) ? Fcar (data) : error_symbol);
(gdb) xbacktrace
"substring" (0xffff6f00)
"string=" (0xffff70a0)
"if" (0xffff7230)
"while" (0xffff7400)
"catch" (0xffff7600)
"let" (0xffff7860)
"and" (0xffff79f0)
"let" (0xffff7c50)
"and" (0xffff7de0)
"or" (0xffff7f70)
"let" (0xffff81d0)
"skkdic-breakup-string" (0xffff8380)
"or" (0xffff8690)
"and" (0xffff8820)
"or" (0xffff89b0)
"if" (0xffff8b40)
"let" (0xffff8da0)
"while" (0xffff8f70)
"let" (0xffff91e0)
"and" (0xffff9370)
"or" (0xffff9500)
"let" (0xffff9760)
"skkdic-breakup-string" (0xffff9910)
"not" (0xffff9be0)
"and" (0xffff9d70)
"or" (0xffff9f00)
"if" (0xffffa090)
"while" (0xffffa260)
"let" (0xffffa4c0)
"skkdic-reduced-candidates" (0xffffa670)
"setq" (0xffffa9b0)
"if" (0xffffab40)
"let" (0xffffada0)
"while" (0xffffaf70)
"let" (0xffffb1e0)
"save-current-buffer" (0xffffb390)
"skkdic-convert-okuri-nasi" (0xffffb540)
"let" (0xffffb900)
"save-current-buffer" (0xffffbab0)
"let*" (0xffffbcc0)
"skkdic-convert" (0xffffbe70)
"let" (0xffffc230)
"if" (0xffffc3f0)
"batch-skkdic-convert" (0xffffc6e8)
"command-line-1" (0xffffcf80)
"command-line" (0xffffd878)
"normal-top-level" (0xffffe040)

(gdb) bt
#0  0x0000000000634e00 in Fsignal (error_symbol=..., data=...) at eval.c:1476
#1  0x00000000006351d9 in xsignal (error_symbol=..., data=...) at eval.c:1582
#2  0x00000000006352ae in xsignal3 (error_symbol=..., arg1=...,
arg2=..., arg3=...) at eval.c:1609
#3  0x0000000000614d16 in args_out_of_range_3 (a1=..., a2=..., a3=...)
at data.c:169
#4  0x000000000063f687 in validate_subarray (array=..., from=...,
to=..., size=4, ifrom=0x7fffffff6e18, ito=0x7fffffff6e10) at
fns.c:1210
#5  0x000000000063f70c in Fsubstring (string=..., from=..., to=...) at
fns.c:1235
#6  0x0000000000636da4 in eval_sub (form=...) at eval.c:2177
#7  0x0000000000636c98 in eval_sub (form=...) at eval.c:2159
#8  0x0000000000631bb4 in Fif (args=...) at eval.c:385
#9  0x00000000006369df in eval_sub (form=...) at eval.c:2124
#10 0x0000000000631e85 in Fprogn (body=...) at eval.c:431

(gdb) info f
Stack level 1, frame at 0x7fffffff6cf0:
 rip = 0x6351d9 in xsignal (eval.c:1582); saved rip = 0x6352ae
 called by frame at 0x7fffffff6d40, caller of frame at 0x7fffffff6cc0
 source language c.
 Arglist at 0x7fffffff6ce0, args: error_symbol=..., data=...
 Locals at 0x7fffffff6ce0, Previous frame's sp is 0x7fffffff6cf0
 Saved registers:
  rbp at 0x7fffffff6ce0, rip at 0x7fffffff6ce8


On 23 October 2016 at 16:48, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Sam Halliday <sam.halliday@gmail.com>
>> Date: Sun, 23 Oct 2016 16:42:20 +0100
>> Cc: npostavs@users.sourceforge.net, 24358@debbugs.gnu.org
>>
>> breaking on Fsignal is hitting the problem... not sure what to do
>> here. I can type "bt" but that's where my gdb skillz end... do you
>> want me to do anything else or is this enough?
>
> "bt" is a good start, thanks.  After that, the results of "xbacktrace"
> will help more.
>
>> I'd imagine seeing the contents of error_symbol and data would be
>> pretty useful. I'm not sure the etc/DEBUG .gdbinit file is being
>> loaded properly when invoked from the leim directory.
>
> You can read .gdbinit manually:
>
>   (gdb) source ../src/.gdbinit
>
> And yes, making sure error_symbol and data are as we expect them would
> be a good start.
>
> Thanks.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Sam Halliday]

aha! Found out how to print the args

(gdb) info args
error_symbol = {
  i = 9600
}
data = {
  i = 116316307
}

(gdb) f 2
#2  0x00000000006352ae in xsignal3 (error_symbol=..., arg1=...,
arg2=..., arg3=...) at eval.c:1609
1609  xsignal (error_symbol, list3 (arg1, arg2, arg3));
(gdb) info args
error_symbol = {
  i = 9600
}
arg1 = {
  i = 63960004
}
arg2 = {
  i = -18
}
arg3 = {
  i = 0
}


On 23 October 2016 at 16:58, Sam Halliday <sam.halliday@gmail.com> wrote:
> ok, that worked to source the .gdbinit and xbacktrace works now...
> (this is a fresh session).
>
> I don't know how to print out the parameters to this function. I tried
> "info" (see below) but it's still giving ...
>
>
> EMACSLOADPATH= gdb ../src/emacs
> GNU gdb (GDB) 7.12
> Copyright (C) 2016 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-pc-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from ../src/emacs...done.
> (gdb) break Fsigna
> Function "Fsigna" not defined.
> Make breakpoint pending on future shared library load? (y or [n]) y
> Breakpoint 1 (Fsigna) pending.
> (gdb) break Fsignal
> Breakpoint 2 at 0x634e00: file eval.c, line 1476.
> (gdb) source ../src/.gdbinit
> SIGINT is used by the debugger.
> Are you sure you want to change it? (y or n) [answered Y; input not
> from terminal]
> DISPLAY = :0
> TERM = rxvt-unicode-256color
> Breakpoint 3 at 0x583686: file emacs.c, line 354.
> Temporary breakpoint 4 at 0x5ae0bc: file sysdep.c, line 915.
> (gdb) run -batch --no-site-file --no-site-lisp -batch -l ja-dic-cnv -f
> batch-skkdic-convert -dir "./../lisp/leim/ja-dic"
> "SKK-DIC/SKK-JISYO.L"
> Starting program: /home/fommil/build/emacs-git/emacs/src/emacs -batch
> --no-site-file --no-site-lisp -batch -l ja-dic-cnv -f
> batch-skkdic-convert -dir "./../lisp/leim/ja-dic"
> "SKK-DIC/SKK-JISYO.L"
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/usr/lib/libthread_db.so.1".
> [New Thread 0x7fffe72e5700 (LWP 24197)]
> Reading file "/home/fommil/build/emacs-git/emacs/leim/SKK-DIC/SKK-JISYO.L" ...
> Processing OKURI-ARI entries ...
> Processing POSTFIX entries ...
> Processing PREFIX entries ...
> Collecting OKURI-NASI entries ...
> collected 26% ...
> collected 30% ...
> collected 40% ...
> collected 50% ...
> collected 60% ...
> collected 70% ...
> collected 80% ...
> collected 90% ...
> Processing OKURI-NASI entries ...
> processed 10% ...
> processed 20% ...
> processed 30% ...
> processed 40% ...
>
> Thread 1 "emacs" hit Breakpoint 2, Fsignal (error_symbol=...,
> data=...) at eval.c:1476
> 1476    = (NILP (error_symbol) ? Fcar (data) : error_symbol);
> (gdb) xbacktrace
> "substring" (0xffff6f00)
> "string=" (0xffff70a0)
> "if" (0xffff7230)
> "while" (0xffff7400)
> "catch" (0xffff7600)
> "let" (0xffff7860)
> "and" (0xffff79f0)
> "let" (0xffff7c50)
> "and" (0xffff7de0)
> "or" (0xffff7f70)
> "let" (0xffff81d0)
> "skkdic-breakup-string" (0xffff8380)
> "or" (0xffff8690)
> "and" (0xffff8820)
> "or" (0xffff89b0)
> "if" (0xffff8b40)
> "let" (0xffff8da0)
> "while" (0xffff8f70)
> "let" (0xffff91e0)
> "and" (0xffff9370)
> "or" (0xffff9500)
> "let" (0xffff9760)
> "skkdic-breakup-string" (0xffff9910)
> "not" (0xffff9be0)
> "and" (0xffff9d70)
> "or" (0xffff9f00)
> "if" (0xffffa090)
> "while" (0xffffa260)
> "let" (0xffffa4c0)
> "skkdic-reduced-candidates" (0xffffa670)
> "setq" (0xffffa9b0)
> "if" (0xffffab40)
> "let" (0xffffada0)
> "while" (0xffffaf70)
> "let" (0xffffb1e0)
> "save-current-buffer" (0xffffb390)
> "skkdic-convert-okuri-nasi" (0xffffb540)
> "let" (0xffffb900)
> "save-current-buffer" (0xffffbab0)
> "let*" (0xffffbcc0)
> "skkdic-convert" (0xffffbe70)
> "let" (0xffffc230)
> "if" (0xffffc3f0)
> "batch-skkdic-convert" (0xffffc6e8)
> "command-line-1" (0xffffcf80)
> "command-line" (0xffffd878)
> "normal-top-level" (0xffffe040)
>
> (gdb) bt
> #0  0x0000000000634e00 in Fsignal (error_symbol=..., data=...) at eval.c:1476
> #1  0x00000000006351d9 in xsignal (error_symbol=..., data=...) at eval.c:1582
> #2  0x00000000006352ae in xsignal3 (error_symbol=..., arg1=...,
> arg2=..., arg3=...) at eval.c:1609
> #3  0x0000000000614d16 in args_out_of_range_3 (a1=..., a2=..., a3=...)
> at data.c:169
> #4  0x000000000063f687 in validate_subarray (array=..., from=...,
> to=..., size=4, ifrom=0x7fffffff6e18, ito=0x7fffffff6e10) at
> fns.c:1210
> #5  0x000000000063f70c in Fsubstring (string=..., from=..., to=...) at
> fns.c:1235
> #6  0x0000000000636da4 in eval_sub (form=...) at eval.c:2177
> #7  0x0000000000636c98 in eval_sub (form=...) at eval.c:2159
> #8  0x0000000000631bb4 in Fif (args=...) at eval.c:385
> #9  0x00000000006369df in eval_sub (form=...) at eval.c:2124
> #10 0x0000000000631e85 in Fprogn (body=...) at eval.c:431
>
> (gdb) info f
> Stack level 1, frame at 0x7fffffff6cf0:
>  rip = 0x6351d9 in xsignal (eval.c:1582); saved rip = 0x6352ae
>  called by frame at 0x7fffffff6d40, caller of frame at 0x7fffffff6cc0
>  source language c.
>  Arglist at 0x7fffffff6ce0, args: error_symbol=..., data=...
>  Locals at 0x7fffffff6ce0, Previous frame's sp is 0x7fffffff6cf0
>  Saved registers:
>   rbp at 0x7fffffff6ce0, rip at 0x7fffffff6ce8
>
>
> On 23 October 2016 at 16:48, Eli Zaretskii <eliz@gnu.org> wrote:
>>> From: Sam Halliday <sam.halliday@gmail.com>
>>> Date: Sun, 23 Oct 2016 16:42:20 +0100
>>> Cc: npostavs@users.sourceforge.net, 24358@debbugs.gnu.org
>>>
>>> breaking on Fsignal is hitting the problem... not sure what to do
>>> here. I can type "bt" but that's where my gdb skillz end... do you
>>> want me to do anything else or is this enough?
>>
>> "bt" is a good start, thanks.  After that, the results of "xbacktrace"
>> will help more.
>>
>>> I'd imagine seeing the contents of error_symbol and data would be
>>> pretty useful. I'm not sure the etc/DEBUG .gdbinit file is being
>>> loaded properly when invoked from the leim directory.
>>
>> You can read .gdbinit manually:
>>
>>   (gdb) source ../src/.gdbinit
>>
>> And yes, making sure error_symbol and data are as we expect them would
>> be a good start.
>>
>> Thanks.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: Sam Halliday <sam.halliday@gmail.com>
> Date: Sun, 23 Oct 2016 16:58:59 +0100
> Cc: npostavs@users.sourceforge.net, 24358@debbugs.gnu.org
> 
> aha! Found out how to print the args
> 
> (gdb) info args
> error_symbol = {
>   i = 9600
> }
> data = {
>   i = 116316307
> }
> 
> (gdb) f 2
> #2  0x00000000006352ae in xsignal3 (error_symbol=..., arg1=...,
> arg2=..., arg3=...) at eval.c:1609
> 1609  xsignal (error_symbol, list3 (arg1, arg2, arg3));
> (gdb) info args
> error_symbol = {
>   i = 9600
> }
> arg1 = {
>   i = 63960004
> }
> arg2 = {
>   i = -18
> }
> arg3 = {
>   i = 0
> }

Something like

  (gdb) p arg1
  (gdb) xpr

will show a human-readable description of the Lisp objects.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: Sam Halliday <sam.halliday@gmail.com>
> Date: Sun, 23 Oct 2016 16:58:07 +0100
> Cc: npostavs@users.sourceforge.net, 24358@debbugs.gnu.org
> 
> Processing OKURI-NASI entries ...
> processed 10% ...
> processed 20% ...
> processed 30% ...
> processed 40% ...
> 
> Thread 1 "emacs" hit Breakpoint 2, Fsignal (error_symbol=...,
> data=...) at eval.c:1476
> 1476    = (NILP (error_symbol) ? Fcar (data) : error_symbol);
> (gdb) xbacktrace
> "substring" (0xffff6f00)
> "string=" (0xffff70a0)
> "if" (0xffff7230)
> "while" (0xffff7400)
> "catch" (0xffff7600)
> "let" (0xffff7860)
> "and" (0xffff79f0)
> "let" (0xffff7c50)
> "and" (0xffff7de0)
> "or" (0xffff7f70)
> "let" (0xffff81d0)
> "skkdic-breakup-string" (0xffff8380)

Thanks.

Does it help to delete this line:

  re_match_object = Qnil;       /* Stop protecting string from GC.  */

in src/search.c:string_match_1 (should be line 404)?

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

Noam, I think we need these two changes, because otherwise looping
more than once in search_buffer will fail to update the pointers
passed to re_search_2, if buffer text was relocated inside
re_search_2.

Do you agree?

diff --git a/src/search.c b/src/search.c
index ec5a1d7..5c04916 100644
--- a/src/search.c
+++ b/src/search.c
@@ -1233,6 +1233,8 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 			      ? &search_regs : &search_regs_1),
 			     /* Don't allow match past current point */
 			     pos_byte - BEGV_BYTE);
+	  /* Update 'base' due to possible relocation inside re_search_2.  */
+	  base = current_buffer->text->beg;
 	  if (val == -2)
 	    {
 	      matcher_overflow ();
@@ -1279,6 +1281,8 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 			     (NILP (Vinhibit_changing_match_data)
 			      ? &search_regs : &search_regs_1),
 			     lim_byte - BEGV_BYTE);
+	  /* Update 'base' due to possible relocation inside re_search_2.  */
+	  base = current_buffer->text->beg;
 	  if (val == -2)
 	    {
 	      matcher_overflow ();

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Noam Postavsky]

On Sun, Oct 23, 2016 at 2:06 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> Noam, I think we need these two changes, because otherwise looping
> more than once in search_buffer will fail to update the pointers
> passed to re_search_2, if buffer text was relocated inside
> re_search_2.
>
> Do you agree?

Ack, yes! Missing the update to base was a total thinko on my part.

>
> diff --git a/src/search.c b/src/search.c
> index ec5a1d7..5c04916 100644
> --- a/src/search.c
> +++ b/src/search.c
> @@ -1233,6 +1233,8 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
>                               ? &search_regs : &search_regs_1),
>                              /* Don't allow match past current point */
>                              pos_byte - BEGV_BYTE);
> +         /* Update 'base' due to possible relocation inside re_search_2.  */
> +         base = current_buffer->text->beg;
>           if (val == -2)
>             {
>               matcher_overflow ();
> @@ -1279,6 +1281,8 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
>                              (NILP (Vinhibit_changing_match_data)
>                               ? &search_regs : &search_regs_1),
>                              lim_byte - BEGV_BYTE);
> +         /* Update 'base' due to possible relocation inside re_search_2.  */
> +         base = current_buffer->text->beg;
>           if (val == -2)
>             {
>               matcher_overflow ();

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Sun, 23 Oct 2016 14:14:59 -0400
> Cc: Sam Halliday <sam.halliday@gmail.com>, 24358@debbugs.gnu.org
> 
> On Sun, Oct 23, 2016 at 2:06 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> > Noam, I think we need these two changes, because otherwise looping
> > more than once in search_buffer will fail to update the pointers
> > passed to re_search_2, if buffer text was relocated inside
> > re_search_2.
> >
> > Do you agree?
> 
> Ack, yes! Missing the update to base was a total thinko on my part.

Pushed.

There might be a more serious problem with this, unfortunately: the
search registers are computed in regex.c using pointers into the C
strings that are being searched.  The general paradigm is as in this
fragment:

  regstart[*p] = d;
  [...]
  regs->start[reg] = POINTER_TO_OFFSET (regstart[reg]);

POINTER_TO_OFFSET assumes that the pointer in regstart[reg] is
consistent with the current base address of the string into which it
points.  Did you study this aspect of regex.c when you decided which
values need to be affected by relocation?

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

I was able to reproduce the failure in ja-dic-cnv after removing
--with-x-toolkit=lucid --without-toolkit-scroll-bars --with-gif=no
--with-jpeg=no from my configure args (I guess using GTK changes the
allocation patterns), and doing 'make extraclean'.

The error went away after I updated to the latest emacs-25 commit:
ee04aedc72 "Fix handling of buffer relocation in regex.c functions".
The error occurs in the commit just before it 71ca4f6a "Avoid relocating
buffers while libxml2 reads its text", so I think Eli's fix for the base
pointer in search_buffer does solve the problem.

Sam, can you confirm if this fixes it for you?

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Noam Postavsky <npostavs@users.sourceforge.net>
>> Date: Sun, 23 Oct 2016 14:14:59 -0400
>> Cc: Sam Halliday <sam.halliday@gmail.com>, 24358@debbugs.gnu.org
>> 
>> On Sun, Oct 23, 2016 at 2:06 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> > Noam, I think we need these two changes, because otherwise looping
>> > more than once in search_buffer will fail to update the pointers
>> > passed to re_search_2, if buffer text was relocated inside
>> > re_search_2.
>> >
>> > Do you agree?
>> 
>> Ack, yes! Missing the update to base was a total thinko on my part.
>
> Pushed.
>
> There might be a more serious problem with this, unfortunately: the
> search registers are computed in regex.c using pointers into the C
> strings that are being searched.  The general paradigm is as in this
> fragment:
>
>   regstart[*p] = d;
>   [...]
>   regs->start[reg] = POINTER_TO_OFFSET (regstart[reg]);
>
> POINTER_TO_OFFSET assumes that the pointer in regstart[reg] is
> consistent with the current base address of the string into which it
> points.  Did you study this aspect of regex.c when you decided which
> values need to be affected by relocation?

I did not look at that before, but looking now, I don't see why it would
be a problem.  I put the base address updating code around the only
place where malloc may be called, so string1 and string2 (which
POINTER_TO_OFFSET uses) should always be consistent with the base
address (unless there is some other malloc call that I missed?).

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: 24358@debbugs.gnu.org,  sam.halliday@gmail.com
> Date: Mon, 24 Oct 2016 09:29:21 -0400
> 
> >   regstart[*p] = d;
> >   [...]
> >   regs->start[reg] = POINTER_TO_OFFSET (regstart[reg]);
> >
> > POINTER_TO_OFFSET assumes that the pointer in regstart[reg] is
> > consistent with the current base address of the string into which it
> > points.  Did you study this aspect of regex.c when you decided which
> > values need to be affected by relocation?
> 
> I did not look at that before, but looking now, I don't see why it would
> be a problem.  I put the base address updating code around the only
> place where malloc may be called, so string1 and string2 (which
> POINTER_TO_OFFSET uses) should always be consistent with the base
> address (unless there is some other malloc call that I missed?).

What bothers me is this: could it be that relocation happens between
the first and the second line above?  If it can, then what
POINTER_TO_OFFSET does will be inconsistent with the base address at
the time regstart[*p] was assigned the value of d.

The code runs in a loop, or so it seems, so it's hard to reason about
time sequences.

But I'm not saying I clearly see a problem, just that I fear there
might be one.  If your reading of the code is that it cannot happen,
I'm happy.

Thanks.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Noam Postavsky]

On Mon, Oct 24, 2016 at 9:39 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: npostavs@users.sourceforge.net
>> Cc: 24358@debbugs.gnu.org,  sam.halliday@gmail.com
>> Date: Mon, 24 Oct 2016 09:29:21 -0400
>>
>> >   regstart[*p] = d;
>> >   [...]
>> >   regs->start[reg] = POINTER_TO_OFFSET (regstart[reg]);
>> >
>> > POINTER_TO_OFFSET assumes that the pointer in regstart[reg] is
>> > consistent with the current base address of the string into which it
>> > points.  Did you study this aspect of regex.c when you decided which
>> > values need to be affected by relocation?
>>
>> I did not look at that before, but looking now, I don't see why it would
>> be a problem.  I put the base address updating code around the only
>> place where malloc may be called, so string1 and string2 (which
>> POINTER_TO_OFFSET uses) should always be consistent with the base
>> address (unless there is some other malloc call that I missed?).
>
> What bothers me is this: could it be that relocation happens between
> the first and the second line above?  If it can, then what
> POINTER_TO_OFFSET does will be inconsistent with the base address at
> the time regstart[*p] was assigned the value of d.
>
> The code runs in a loop, or so it seems, so it's hard to reason about
> time sequences.

Oh, I see. Yes, I think you're right, the pointers stored in regstart,
regend, and fail_stack could become inconsistent. Hard to say what
kind of regex could trigger it, but it seems quite possible.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Mon, 24 Oct 2016 11:33:11 -0400
> Cc: 24358@debbugs.gnu.org, Sam Halliday <sam.halliday@gmail.com>
> 
> > What bothers me is this: could it be that relocation happens between
> > the first and the second line above?  If it can, then what
> > POINTER_TO_OFFSET does will be inconsistent with the base address at
> > the time regstart[*p] was assigned the value of d.
> >
> > The code runs in a loop, or so it seems, so it's hard to reason about
> > time sequences.
> 
> Oh, I see. Yes, I think you're right, the pointers stored in regstart,
> regend, and fail_stack could become inconsistent. Hard to say what
> kind of regex could trigger it, but it seems quite possible.

Maybe we should simply call r_alloc_inhibit_buffer_relocation around
any calls to regex.c functions, and remove the code you wrote to
handle relocations internally.

I'm sorry I didn't look at this closer before, and thus caused you to
work on all those changes, which we might now need to revert.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

[-- Attachment #1: Type: text/plain, Size: 432 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:
>> Oh, I see. Yes, I think you're right, the pointers stored in regstart,
>> regend, and fail_stack could become inconsistent. Hard to say what
>> kind of regex could trigger it, but it seems quite possible.
>
> Maybe we should simply call r_alloc_inhibit_buffer_relocation around
> any calls to regex.c functions, and remove the code you wrote to
> handle relocations internally.

How's this:


[-- Attachment #2: reverting patch --]
[-- Type: text/plain, Size: 12015 bytes --]

From c3e87ee1e1399f3687db02dc71b13830852a50eb Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Mon, 24 Oct 2016 19:54:29 -0400
Subject: [PATCH v4 1/2] Revert fixes to allocation of regex matching

The fix was not complete, and completing it was proving too complicated.

- Revert "* src/regex.c (re_search_2): Make new code safe for -Wjump-misses-init."
  This reverts commit c2a17924a57483d14692c8913edbe8ad24b5ffbb.
- Revert "Port to GCC 6.2.1 + --enable-gcc-warnings"
  This reverts commit f6134bbda259c115c06d4a9a3ab5c39340a15949.
- Revert "Fix handling of allocation in regex matching"
  This reverts commit ad66b3fadb7ae22a4cbb82bb1507c39ceadf3897.
- Revert "Fix handling of buffer relocation in regex.c functions"
  This reverts commit ee04aedc723b035eedaf975422d4eb242894121b.
---
 src/dired.c  |  4 +---
 src/regex.c  | 73 ------------------------------------------------------------
 src/regex.h  |  4 +---
 src/search.c | 40 ++++++++++-----------------------
 4 files changed, 14 insertions(+), 107 deletions(-)

diff --git a/src/dired.c b/src/dired.c
index 006f74c..dba575c 100644
--- a/src/dired.c
+++ b/src/dired.c
@@ -259,11 +259,9 @@ directory_files_internal (Lisp_Object directory, Lisp_Object full,
       QUIT;
 
       bool wanted = (NILP (match)
-		     || (re_match_object = name,
-                         re_search (bufp, SSDATA (name), len, 0, len, 0) >= 0));
+		     || re_search (bufp, SSDATA (name), len, 0, len, 0) >= 0);
 
       immediate_quit = 0;
-      re_match_object = Qnil;   /* Stop protecting name from GC.  */
 
       if (wanted)
 	{
diff --git a/src/regex.c b/src/regex.c
index b12e95b..56b18e6 100644
--- a/src/regex.c
+++ b/src/regex.c
@@ -1438,62 +1438,11 @@ WEAK_ALIAS (__re_set_syntax, re_set_syntax)
 #define NEXT_FAILURE_HANDLE(h) fail_stack.stack[(h) - 3].integer
 #define TOP_FAILURE_HANDLE() fail_stack.frame
 
-#ifdef emacs
-# define STR_BASE_PTR(obj)			 \
-   (NILP (obj) ? current_buffer->text->beg	 \
-    : STRINGP (obj) ? SDATA (obj)		 \
-    : NULL)
-#else
-# define STR_BASE_PTR(obj) NULL
-#endif
 
 #define ENSURE_FAIL_STACK(space)					\
 while (REMAINING_AVAIL_SLOTS <= space) {				\
-  re_char *orig_base = STR_BASE_PTR (re_match_object);                  \
-  bool might_relocate = orig_base != NULL;				\
-  ptrdiff_t string1_off, end1_off, end_match_1_off;                     \
-  ptrdiff_t string2_off, end2_off, end_match_2_off;                     \
-  ptrdiff_t d_off, dend_off, dfail_off;                                 \
-  if (might_relocate)							\
-    {                                                                   \
-      if (string1)                                                      \
-        {                                                               \
-          string1_off = string1 - orig_base;                            \
-          end1_off = end1 - orig_base;                                  \
-          end_match_1_off = end_match_1 - orig_base;                    \
-        }                                                               \
-      if (string2)                                                      \
-        {                                                               \
-          string2_off = string2 - orig_base;                            \
-          end2_off = end2 - orig_base;                                  \
-          end_match_2_off = end_match_2 - orig_base;                    \
-        }                                                               \
-      d_off = d - orig_base;                                            \
-      dend_off = dend - orig_base;                                      \
-      dfail_off = dfail - orig_base;                                    \
-    }                                                                   \
   if (!GROW_FAIL_STACK (fail_stack))					\
     return -2;								\
-  /* In Emacs, GROW_FAIL_STACK might relocate string pointers.  */	\
-  if (might_relocate)							\
-    {                                                                   \
-      re_char *new_base = STR_BASE_PTR (re_match_object);		\
-      if (string1)                                                      \
-        {                                                               \
-          string1 = new_base + string1_off;                             \
-          end1 = new_base + end1_off;                                   \
-          end_match_1 = new_base + end_match_1_off;                     \
-        }                                                               \
-      if (string2)                                                      \
-        {                                                               \
-          string2 = new_base + string2_off;                             \
-          end2 = new_base + end2_off;                                   \
-          end_match_2 = new_base + end_match_2_off;                     \
-        }                                                               \
-      d = new_base + d_off;                                             \
-      dend = new_base + dend_off;                                       \
-      dfail = new_base + dfail_off;                                     \
-    }                                                                   \
   DEBUG_PRINT ("\n  Doubled stack; size now: %zd\n", (fail_stack).size);\
   DEBUG_PRINT ("	 slots available: %zd\n", REMAINING_AVAIL_SLOTS);\
 }
@@ -4380,10 +4329,6 @@ re_search_2 (struct re_pattern_buffer *bufp, const char *str1, size_t size1,
   /* Loop through the string, looking for a place to start matching.  */
   for (;;)
     {
-      ptrdiff_t offset1, offset2;
-      re_char *orig_base;
-      bool might_relocate;
-
       /* If the pattern is anchored,
 	 skip quickly past places we cannot match.
 	 We don't bother to treat startpos == 0 specially
@@ -4500,17 +4445,6 @@ re_search_2 (struct re_pattern_buffer *bufp, const char *str1, size_t size1,
 	  && !bufp->can_be_null)
 	return -1;
 
-      /* re_match_2_internal may allocate, relocating the Lisp text
-	 object that we're searching.  */
-      IF_LINT (offset2 = 0);  /* Work around GCC bug 78081.  */
-      orig_base = STR_BASE_PTR (re_match_object);
-      might_relocate = orig_base != NULL;
-      if (might_relocate)
-        {
-          if (string1) offset1 = string1 - orig_base;
-          if (string2) offset2 = string2 - orig_base;
-        }
-
       val = re_match_2_internal (bufp, string1, size1, string2, size2,
 				 startpos, regs, stop);
 
@@ -4520,13 +4454,6 @@ re_search_2 (struct re_pattern_buffer *bufp, const char *str1, size_t size1,
       if (val == -2)
 	return -2;
 
-      if (might_relocate)
-        {
-	  re_char *new_base = STR_BASE_PTR (re_match_object);
-          if (string1) string1 = offset1 + new_base;
-          if (string2) string2 = offset2 + new_base;
-        }
-
     advance:
       if (!range)
 	break;
diff --git a/src/regex.h b/src/regex.h
index 61c771c..51f4424 100644
--- a/src/regex.h
+++ b/src/regex.h
@@ -169,9 +169,7 @@ extern reg_syntax_t re_syntax_options;
 #ifdef emacs
 # include "lisp.h"
 /* In Emacs, this is the string or buffer in which we are matching.
-   It is used for looking up syntax properties, and also to recompute
-   pointers in case the object is relocated as a side effect of
-   calling malloc (if it calls r_alloc_sbrk in ralloc.c).
+   It is used for looking up syntax properties.
 
    If the value is a Lisp string object, we are matching text in that
    string; if it's nil, we are matching text in the current buffer; if
diff --git a/src/search.c b/src/search.c
index b50e7f0..fa5ac44 100644
--- a/src/search.c
+++ b/src/search.c
@@ -287,10 +287,8 @@ looking_at_1 (Lisp_Object string, bool posix)
   immediate_quit = 1;
   QUIT;			/* Do a pending quit right away, to avoid paradoxical behavior */
 
-  /* Get pointers and sizes of the two strings that make up the
-     visible portion of the buffer.  Note that we can use pointers
-     here, unlike in search_buffer, because we only call re_match_2
-     once, after which we never use the pointers again.  */
+  /* Get pointers and sizes of the two strings
+     that make up the visible portion of the buffer. */
 
   p1 = BEGV_ADDR;
   s1 = GPT_BYTE - BEGV_BYTE;
@@ -409,7 +407,6 @@ string_match_1 (Lisp_Object regexp, Lisp_Object string, Lisp_Object start,
 		   (NILP (Vinhibit_changing_match_data)
 		    ? &search_regs : NULL));
   immediate_quit = 0;
-  re_match_object = Qnil;       /* Stop protecting string from GC.  */
 
   /* Set last_thing_searched only when match data is changed.  */
   if (NILP (Vinhibit_changing_match_data))
@@ -480,7 +477,6 @@ fast_string_match_internal (Lisp_Object regexp, Lisp_Object string,
 		   SBYTES (string), 0,
 		   SBYTES (string), 0);
   immediate_quit = 0;
-  re_match_object = Qnil;       /* Stop protecting string from GC.  */
   return val;
 }
 
@@ -568,7 +564,6 @@ fast_looking_at (Lisp_Object regexp, ptrdiff_t pos, ptrdiff_t pos_byte,
   len = re_match_2 (buf, (char *) p1, s1, (char *) p2, s2,
 		    pos_byte, NULL, limit_byte);
   immediate_quit = 0;
-  re_match_object = Qnil;       /* Stop protecting string from GC.  */
 
   return len;
 }
@@ -1183,8 +1178,8 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 
   if (RE && !(trivial_regexp_p (string) && NILP (Vsearch_spaces_regexp)))
     {
-      unsigned char *base;
-      ptrdiff_t off1, off2, s1, s2;
+      unsigned char *p1, *p2;
+      ptrdiff_t s1, s2;
       struct re_pattern_buffer *bufp;
 
       bufp = compile_pattern (string,
@@ -1198,19 +1193,16 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 				   can take too long. */
       QUIT;			/* Do a pending quit right away,
 				   to avoid paradoxical behavior */
-      /* Get offsets and sizes of the two strings that make up the
-         visible portion of the buffer.  We compute offsets instead of
-         pointers because re_search_2 may call malloc and therefore
-         change the buffer text address.  */
+      /* Get pointers and sizes of the two strings
+	 that make up the visible portion of the buffer. */
 
-      base = current_buffer->text->beg;
-      off1 = BEGV_ADDR - base;
+      p1 = BEGV_ADDR;
       s1 = GPT_BYTE - BEGV_BYTE;
-      off2 = GAP_END_ADDR - base;
+      p2 = GAP_END_ADDR;
       s2 = ZV_BYTE - GPT_BYTE;
       if (s1 < 0)
 	{
-          off2 = off1;
+	  p2 = p1;
 	  s2 = ZV_BYTE - BEGV_BYTE;
 	  s1 = 0;
 	}
@@ -1225,16 +1217,12 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 	{
 	  ptrdiff_t val;
 
-          val = re_search_2 (bufp,
-                             (char*) (base + off1), s1,
-                             (char*) (base + off2), s2,
+	  val = re_search_2 (bufp, (char *) p1, s1, (char *) p2, s2,
 			     pos_byte - BEGV_BYTE, lim_byte - pos_byte,
 			     (NILP (Vinhibit_changing_match_data)
 			      ? &search_regs : &search_regs_1),
 			     /* Don't allow match past current point */
 			     pos_byte - BEGV_BYTE);
-	  /* Update 'base' due to possible relocation inside re_search_2.  */
-	  base = current_buffer->text->beg;
 	  if (val == -2)
 	    {
 	      matcher_overflow ();
@@ -1274,15 +1262,11 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 	{
 	  ptrdiff_t val;
 
-          val = re_search_2 (bufp,
-                             (char*) (base + off1), s1,
-                             (char*) (base + off2), s2,
-                             pos_byte - BEGV_BYTE, lim_byte - pos_byte,
+	  val = re_search_2 (bufp, (char *) p1, s1, (char *) p2, s2,
+			     pos_byte - BEGV_BYTE, lim_byte - pos_byte,
 			     (NILP (Vinhibit_changing_match_data)
 			      ? &search_regs : &search_regs_1),
 			     lim_byte - BEGV_BYTE);
-	  /* Update 'base' due to possible relocation inside re_search_2.  */
-	  base = current_buffer->text->beg;
 	  if (val == -2)
 	    {
 	      matcher_overflow ();
-- 
2.9.3


[-- Attachment #3: patch --]
[-- Type: text/plain, Size: 2823 bytes --]

From 79806c03ad9f45d8ed016a06f5e24cf69df85ad2 Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Mon, 24 Oct 2016 21:22:07 -0400
Subject: [PATCH v4 2/2] Inhibit buffer relocation during regex searches

* src/search.c (looking_at_1, fast_looking_at, search_buffer): Prevent
relocation of buffer contents during calls to re_search_2.  This ensures
the pointers into buffer text won't be invalidated by
r_alloc_sbrk (called from malloc with configurations where
REL_ALLOC=yes).
---
 src/search.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/src/search.c b/src/search.c
index fa5ac44..15504be 100644
--- a/src/search.c
+++ b/src/search.c
@@ -308,12 +308,20 @@ looking_at_1 (Lisp_Object string, bool posix)
 
   re_match_object = Qnil;
 
+#ifdef REL_ALLOC
+  /* Prevent ralloc.c from relocating the current buffer while
+     searching it.  */
+  r_alloc_inhibit_buffer_relocation (1);
+#endif
   i = re_match_2 (bufp, (char *) p1, s1, (char *) p2, s2,
 		  PT_BYTE - BEGV_BYTE,
 		  (NILP (Vinhibit_changing_match_data)
 		   ? &search_regs : NULL),
 		  ZV_BYTE - BEGV_BYTE);
   immediate_quit = 0;
+#ifdef REL_ALLOC
+  r_alloc_inhibit_buffer_relocation (0);
+#endif
 
   if (i == -2)
     matcher_overflow ();
@@ -561,8 +569,16 @@ fast_looking_at (Lisp_Object regexp, ptrdiff_t pos, ptrdiff_t pos_byte,
 
   buf = compile_pattern (regexp, 0, Qnil, 0, multibyte);
   immediate_quit = 1;
+#ifdef REL_ALLOC
+  /* Prevent ralloc.c from relocating the current buffer while
+     searching it.  */
+  r_alloc_inhibit_buffer_relocation (1);
+#endif
   len = re_match_2 (buf, (char *) p1, s1, (char *) p2, s2,
 		    pos_byte, NULL, limit_byte);
+#ifdef REL_ALLOC
+  r_alloc_inhibit_buffer_relocation (0);
+#endif
   immediate_quit = 0;
 
   return len;
@@ -1213,6 +1229,12 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 	}
       re_match_object = Qnil;
 
+#ifdef REL_ALLOC
+  /* Prevent ralloc.c from relocating the current buffer while
+     searching it.  */
+  r_alloc_inhibit_buffer_relocation (1);
+#endif
+
       while (n < 0)
 	{
 	  ptrdiff_t val;
@@ -1254,6 +1276,9 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 	  else
 	    {
 	      immediate_quit = 0;
+#ifdef REL_ALLOC
+              r_alloc_inhibit_buffer_relocation (0);
+#endif
 	      return (n);
 	    }
 	  n++;
@@ -1296,11 +1321,17 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 	  else
 	    {
 	      immediate_quit = 0;
+#ifdef REL_ALLOC
+              r_alloc_inhibit_buffer_relocation (0);
+#endif
 	      return (0 - n);
 	    }
 	  n--;
 	}
       immediate_quit = 0;
+#ifdef REL_ALLOC
+      r_alloc_inhibit_buffer_relocation (0);
+#endif
       return (pos);
     }
   else				/* non-RE case */
-- 
2.9.3


[-- Attachment #4: Type: text/plain, Size: 336 bytes --]


I did 'make extraclean && . ../config-debug.sh && make' (all in one, to
ensure I didn't forget to clean) and it succeeded.

>
> I'm sorry I didn't look at this closer before, and thus caused you to
> work on all those changes, which we might now need to revert.

Yes, that's how hindsight works, you only get it when it's too late. ;)

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: 24358@debbugs.gnu.org,  sam.halliday@gmail.com
> Date: Mon, 24 Oct 2016 22:00:02 -0400
> 
> > Maybe we should simply call r_alloc_inhibit_buffer_relocation around
> > any calls to regex.c functions, and remove the code you wrote to
> > handle relocations internally.
> 
> How's this:

Looks OK, please push.

> I did 'make extraclean && . ../config-debug.sh && make' (all in one, to
> ensure I didn't forget to clean) and it succeeded.

That's good, let's hope Michael's problems disappear as well.

> > I'm sorry I didn't look at this closer before, and thus caused you to
> > work on all those changes, which we might now need to revert.
> 
> Yes, that's how hindsight works, you only get it when it's too late. ;)

;-)

Thanks.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: npostavs@users.sourceforge.net
>> Cc: 24358@debbugs.gnu.org,  sam.halliday@gmail.com
>> Date: Mon, 24 Oct 2016 22:00:02 -0400
>> 
>> > Maybe we should simply call r_alloc_inhibit_buffer_relocation around
>> > any calls to regex.c functions, and remove the code you wrote to
>> > handle relocations internally.
>> 
>> How's this:
>
> Looks OK, please push.

Pushed as 43986d1, fee4cef.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: npostavs@users.sourceforge.net
> Cc: 24358@debbugs.gnu.org,  sam.halliday@gmail.com
> Date: Mon, 24 Oct 2016 09:29:21 -0400
> 
> The error went away after I updated to the latest emacs-25 commit:
> ee04aedc72 "Fix handling of buffer relocation in regex.c functions".
> The error occurs in the commit just before it 71ca4f6a "Avoid relocating
> buffers while libxml2 reads its text", so I think Eli's fix for the base
> pointer in search_buffer does solve the problem.

Good to know, thanks.

> Sam, can you confirm if this fixes it for you?

Fingers crossed.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> Date: Mon, 24 Oct 2016 16:43:31 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: sam.halliday@gmail.com, 24358@debbugs.gnu.org
> 
> > From: npostavs@users.sourceforge.net
> > Cc: 24358@debbugs.gnu.org,  sam.halliday@gmail.com
> > Date: Mon, 24 Oct 2016 09:29:21 -0400
> > 
> > The error went away after I updated to the latest emacs-25 commit:
> > ee04aedc72 "Fix handling of buffer relocation in regex.c functions".
> > The error occurs in the commit just before it 71ca4f6a "Avoid relocating
> > buffers while libxml2 reads its text", so I think Eli's fix for the base
> > pointer in search_buffer does solve the problem.
> 
> Good to know, thanks.

Meanwhile, I fixed yet another potential problem of this class, in
replace-match.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Sam Halliday]

I built 96ac0c3 and got a problem with the same part of the build, but
this time at 10%. Apologies, I don't have the free time to do another
debug run on it tonight.

I tried with REL_ALLOC=no and that builds fine! Thank you for the
workaround. I will still try to help with debug info.


Re: docker, it is libre software and you should be able to install it
using whatever package manager you have. It allows you to run
containerised GNU distributions using your current linux kernel.
Pre-built images are made available through the "hocker hub", e.g.
https://hub.docker.com/r/base/archlinux/

If you have docker installed you would type something like this (don't
forget to add your user to the docker group)

  docker run -it base/archlinux

and (after downloading the "layers"), it will drop you into a shell
running that OS. ArchLinux uses "pacman" as the package manager and
you might need to run something like "pacman -S devel-base emacs" to
get everything you need. Once you exit this shell, all your changes
are lost, so you might want to investigate mounting a local directory
into the container, e.g. so you don't need to download emacs git tree
every time.

I'm no expert, I've only used it a few times before but it is
invaluable when it works.


On 24 October 2016 at 14:29,  <npostavs@users.sourceforge.net> wrote:
> I was able to reproduce the failure in ja-dic-cnv after removing
> --with-x-toolkit=lucid --without-toolkit-scroll-bars --with-gif=no
> --with-jpeg=no from my configure args (I guess using GTK changes the
> allocation patterns), and doing 'make extraclean'.
>
> The error went away after I updated to the latest emacs-25 commit:
> ee04aedc72 "Fix handling of buffer relocation in regex.c functions".
> The error occurs in the commit just before it 71ca4f6a "Avoid relocating
> buffers while libxml2 reads its text", so I think Eli's fix for the base
> pointer in search_buffer does solve the problem.
>
> Sam, can you confirm if this fixes it for you?
>
> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Noam Postavsky <npostavs@users.sourceforge.net>
>>> Date: Sun, 23 Oct 2016 14:14:59 -0400
>>> Cc: Sam Halliday <sam.halliday@gmail.com>, 24358@debbugs.gnu.org
>>>
>>> On Sun, Oct 23, 2016 at 2:06 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> > Noam, I think we need these two changes, because otherwise looping
>>> > more than once in search_buffer will fail to update the pointers
>>> > passed to re_search_2, if buffer text was relocated inside
>>> > re_search_2.
>>> >
>>> > Do you agree?
>>>
>>> Ack, yes! Missing the update to base was a total thinko on my part.
>>
>> Pushed.
>>
>> There might be a more serious problem with this, unfortunately: the
>> search registers are computed in regex.c using pointers into the C
>> strings that are being searched.  The general paradigm is as in this
>> fragment:
>>
>>   regstart[*p] = d;
>>   [...]
>>   regs->start[reg] = POINTER_TO_OFFSET (regstart[reg]);
>>
>> POINTER_TO_OFFSET assumes that the pointer in regstart[reg] is
>> consistent with the current base address of the string into which it
>> points.  Did you study this aspect of regex.c when you decided which
>> values need to be affected by relocation?
>
> I did not look at that before, but looking now, I don't see why it would
> be a problem.  I put the base address updating code around the only
> place where malloc may be called, so string1 and string2 (which
> POINTER_TO_OFFSET uses) should always be consistent with the base
> address (unless there is some other malloc call that I missed?).

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[npostavs]

Sam Halliday <sam.halliday@gmail.com> writes:

> I built 96ac0c3 and got a problem with the same part of the build, but
> this time at 10%. Apologies, I don't have the free time to do another
> debug run on it tonight.

Ah, I can reproduce this.

>
>
> On 24 October 2016 at 14:29,  <npostavs@users.sourceforge.net> wrote:
>> I was able to reproduce the failure in ja-dic-cnv after removing
>> --with-x-toolkit=lucid --without-toolkit-scroll-bars --with-gif=no
>> --with-jpeg=no from my configure args (I guess using GTK changes the
>> allocation patterns), and doing 'make extraclean'.
>>
>> The error went away after I updated to the latest emacs-25 commit:
>> ee04aedc72 "Fix handling of buffer relocation in regex.c functions".

And trying again with ee04aedc72 it fails as well.  I must have forgot
to clean when running it before.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: Sam Halliday <sam.halliday@gmail.com>
> Date: Mon, 24 Oct 2016 21:13:33 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, 24358@debbugs.gnu.org
> 
> I built 96ac0c3 and got a problem with the same part of the build, but
> this time at 10%. Apologies, I don't have the free time to do another
> debug run on it tonight.
> 
> I tried with REL_ALLOC=no and that builds fine! Thank you for the
> workaround. I will still try to help with debug info.

Is there anything else we should do about this bug?  Or can it be
closed?

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Noam Postavsky]

On Sun, Nov 6, 2016 at 10:39 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Sam Halliday <sam.halliday@gmail.com>
>> Date: Mon, 24 Oct 2016 21:13:33 +0100
>> Cc: Eli Zaretskii <eliz@gnu.org>, 24358@debbugs.gnu.org
>>
>> I built 96ac0c3 and got a problem with the same part of the build, but
>> this time at 10%. Apologies, I don't have the free time to do another
>> debug run on it tonight.
>>
>> I tried with REL_ALLOC=no and that builds fine! Thank you for the
>> workaround. I will still try to help with debug info.
>
> Is there anything else we should do about this bug?  Or can it be
> closed?

Actually, it was never officially reopened when Sam reported the
problems with the first patch.

There have been 2 more independent reports that applying this (and the
other rel_alloc related fixes) to 25.1 solves the crashing: bug #24821
(which I've merged to this one), and
https://lists.gnu.org/archive/html/emacs-devel/2016-10/msg00917.html,
so I think we're good.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Sun, 6 Nov 2016 22:56:18 -0500
> Cc: Sam Halliday <sam.halliday@gmail.com>, 24358@debbugs.gnu.org
> 
> > Is there anything else we should do about this bug?  Or can it be
> > closed?
> 
> Actually, it was never officially reopened when Sam reported the
> problems with the first patch.
> 
> There have been 2 more independent reports that applying this (and the
> other rel_alloc related fixes) to 25.1 solves the crashing: bug #24821
> (which I've merged to this one), and
> https://lists.gnu.org/archive/html/emacs-devel/2016-10/msg00917.html,
> so I think we're good.

Great, thanks.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Sam Halliday]

Your patch suggestion results in a failure at 20% in the test for me.

I wish we knew what was causing this to fail on my OS but not yours.
Do you have docker? You could perhaps try with an recent archlinux
image.

On 23 October 2016 at 19:06, Eli Zaretskii <eliz@gnu.org> wrote:
> Noam, I think we need these two changes, because otherwise looping
> more than once in search_buffer will fail to update the pointers
> passed to re_search_2, if buffer text was relocated inside
> re_search_2.
>
> Do you agree?
>
> diff --git a/src/search.c b/src/search.c
> index ec5a1d7..5c04916 100644
> --- a/src/search.c
> +++ b/src/search.c
> @@ -1233,6 +1233,8 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
>                               ? &search_regs : &search_regs_1),
>                              /* Don't allow match past current point */
>                              pos_byte - BEGV_BYTE);
> +         /* Update 'base' due to possible relocation inside re_search_2.  */
> +         base = current_buffer->text->beg;
>           if (val == -2)
>             {
>               matcher_overflow ();
> @@ -1279,6 +1281,8 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
>                              (NILP (Vinhibit_changing_match_data)
>                               ? &search_regs : &search_regs_1),
>                              lim_byte - BEGV_BYTE);
> +         /* Update 'base' due to possible relocation inside re_search_2.  */
> +         base = current_buffer->text->beg;
>           if (val == -2)
>             {
>               matcher_overflow ();

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: Sam Halliday <sam.halliday@gmail.com>
> Date: Sun, 23 Oct 2016 19:16:18 +0100
> Cc: npostavs@users.sourceforge.net, 24358@debbugs.gnu.org
> 
> Your patch suggestion results in a failure at 20% in the test for me.

So none of the 2 patches work.  Which doesn't surprise me: the first
was a stab in the dark anyway, and the second is unrelated to the
problem you reported.

> I wish we knew what was causing this to fail on my OS but not yours.

My build doesn't use ralloc.c.  It's that simple.

> You could perhaps try with an recent archlinux image.

What for?  The problem is clear from your last Lisp backtrace: a call
to string-match returns bad data, which then barfs when used in the
following call to substring.

The problem is to find the code which causes this.  Hmm...

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> Date: Sun, 23 Oct 2016 22:10:55 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 24358@debbugs.gnu.org, npostavs@users.sourceforge.net
> 
> The problem is to find the code which causes this.  Hmm...

Sam, the patch below removes the changes done by Noam in an attempt to
solve the original report of this bug.  Please apply it, and see if
doing so allows you to complete your bootstrap without any errors.

Noam, do you configure Emacs with --enable-check-lisp-object-type?
If not, please do, because that's what Sam does; perhaps that is the
reason why he gets the problem and you don't.

One other difference is that Sam is running a bootstrap, so his
bootstrap-emacs used to build ja-dic has the necessary Lisp files
loaded as *.el files, not *.elc.  Therefore another thing to try
(perhaps even before reconfiguring) is a full bootstrap, preferably in
a separate clean directory, maybe you will see this problem then.

Your systems seem to be similar, so I'd expect you both to see the
same problems.

Here's the patch to remove Noam's changes and Paul's follow-up
changes:

diff --git a/src/dired.c b/src/dired.c
index 006f74c..dba575c 100644
--- a/src/dired.c
+++ b/src/dired.c
@@ -259,11 +259,9 @@ directory_files_internal (Lisp_Object directory, Lisp_Object full,
       QUIT;
 
       bool wanted = (NILP (match)
-		     || (re_match_object = name,
-                         re_search (bufp, SSDATA (name), len, 0, len, 0) >= 0));
+		     || re_search (bufp, SSDATA (name), len, 0, len, 0) >= 0);
 
       immediate_quit = 0;
-      re_match_object = Qnil;   /* Stop protecting name from GC.  */
 
       if (wanted)
 	{
diff --git a/src/regex.c b/src/regex.c
index b12e95b..164eb46 100644
--- a/src/regex.c
+++ b/src/regex.c
@@ -152,8 +152,6 @@
 
 /* Converts the pointer to the char to BEG-based offset from the start.  */
 # define PTR_TO_OFFSET(d) POS_AS_IN_BUFFER (POINTER_TO_OFFSET (d))
-/* Strings are 0-indexed, buffers are 1-indexed; we pun on the boolean
-   result to get the right base index.  */
 # define POS_AS_IN_BUFFER(p) ((p) + (NILP (re_match_object) || BUFFERP (re_match_object)))
 
 # define RE_MULTIBYTE_P(bufp) ((bufp)->multibyte)
@@ -1438,62 +1436,11 @@ typedef struct
 #define NEXT_FAILURE_HANDLE(h) fail_stack.stack[(h) - 3].integer
 #define TOP_FAILURE_HANDLE() fail_stack.frame
 
-#ifdef emacs
-# define STR_BASE_PTR(obj)			 \
-   (NILP (obj) ? current_buffer->text->beg	 \
-    : STRINGP (obj) ? SDATA (obj)		 \
-    : NULL)
-#else
-# define STR_BASE_PTR(obj) NULL
-#endif
 
 #define ENSURE_FAIL_STACK(space)					\
 while (REMAINING_AVAIL_SLOTS <= space) {				\
-  re_char *orig_base = STR_BASE_PTR (re_match_object);                  \
-  bool might_relocate = orig_base != NULL;				\
-  ptrdiff_t string1_off, end1_off, end_match_1_off;                     \
-  ptrdiff_t string2_off, end2_off, end_match_2_off;                     \
-  ptrdiff_t d_off, dend_off, dfail_off;                                 \
-  if (might_relocate)							\
-    {                                                                   \
-      if (string1)                                                      \
-        {                                                               \
-          string1_off = string1 - orig_base;                            \
-          end1_off = end1 - orig_base;                                  \
-          end_match_1_off = end_match_1 - orig_base;                    \
-        }                                                               \
-      if (string2)                                                      \
-        {                                                               \
-          string2_off = string2 - orig_base;                            \
-          end2_off = end2 - orig_base;                                  \
-          end_match_2_off = end_match_2 - orig_base;                    \
-        }                                                               \
-      d_off = d - orig_base;                                            \
-      dend_off = dend - orig_base;                                      \
-      dfail_off = dfail - orig_base;                                    \
-    }                                                                   \
   if (!GROW_FAIL_STACK (fail_stack))					\
     return -2;								\
-  /* In Emacs, GROW_FAIL_STACK might relocate string pointers.  */	\
-  if (might_relocate)							\
-    {                                                                   \
-      re_char *new_base = STR_BASE_PTR (re_match_object);		\
-      if (string1)                                                      \
-        {                                                               \
-          string1 = new_base + string1_off;                             \
-          end1 = new_base + end1_off;                                   \
-          end_match_1 = new_base + end_match_1_off;                     \
-        }                                                               \
-      if (string2)                                                      \
-        {                                                               \
-          string2 = new_base + string2_off;                             \
-          end2 = new_base + end2_off;                                   \
-          end_match_2 = new_base + end_match_2_off;                     \
-        }                                                               \
-      d = new_base + d_off;                                             \
-      dend = new_base + dend_off;                                       \
-      dfail = new_base + dfail_off;                                     \
-    }                                                                   \
   DEBUG_PRINT ("\n  Doubled stack; size now: %zd\n", (fail_stack).size);\
   DEBUG_PRINT ("	 slots available: %zd\n", REMAINING_AVAIL_SLOTS);\
 }
@@ -4380,10 +4327,6 @@ re_search_2 (struct re_pattern_buffer *bufp, const char *str1, size_t size1,
   /* Loop through the string, looking for a place to start matching.  */
   for (;;)
     {
-      ptrdiff_t offset1, offset2;
-      re_char *orig_base;
-      bool might_relocate;
-
       /* If the pattern is anchored,
 	 skip quickly past places we cannot match.
 	 We don't bother to treat startpos == 0 specially
@@ -4500,17 +4443,6 @@ re_search_2 (struct re_pattern_buffer *bufp, const char *str1, size_t size1,
 	  && !bufp->can_be_null)
 	return -1;
 
-      /* re_match_2_internal may allocate, relocating the Lisp text
-	 object that we're searching.  */
-      IF_LINT (offset2 = 0);  /* Work around GCC bug 78081.  */
-      orig_base = STR_BASE_PTR (re_match_object);
-      might_relocate = orig_base != NULL;
-      if (might_relocate)
-        {
-          if (string1) offset1 = string1 - orig_base;
-          if (string2) offset2 = string2 - orig_base;
-        }
-
       val = re_match_2_internal (bufp, string1, size1, string2, size2,
 				 startpos, regs, stop);
 
@@ -4520,13 +4452,6 @@ re_search_2 (struct re_pattern_buffer *bufp, const char *str1, size_t size1,
       if (val == -2)
 	return -2;
 
-      if (might_relocate)
-        {
-	  re_char *new_base = STR_BASE_PTR (re_match_object);
-          if (string1) string1 = offset1 + new_base;
-          if (string2) string2 = offset2 + new_base;
-        }
-
     advance:
       if (!range)
 	break;
@@ -4962,8 +4887,8 @@ WEAK_ALIAS (__re_match, re_match)
 #endif /* not emacs */
 
 #ifdef emacs
-/* In Emacs, this is the string or buffer in which we are matching.
-   See the declaration in regex.h for details.  */
+/* In Emacs, this is the string or buffer in which we
+   are matching.  It is used for looking up syntax properties.  */
 Lisp_Object re_match_object;
 #endif
 
diff --git a/src/search.c b/src/search.c
index 5c04916..dc7e2d8 100644
--- a/src/search.c
+++ b/src/search.c
@@ -287,10 +287,8 @@ looking_at_1 (Lisp_Object string, bool posix)
   immediate_quit = 1;
   QUIT;			/* Do a pending quit right away, to avoid paradoxical behavior */
 
-  /* Get pointers and sizes of the two strings that make up the
-     visible portion of the buffer.  Note that we can use pointers
-     here, unlike in search_buffer, because we only call re_match_2
-     once, after which we never use the pointers again.  */
+  /* Get pointers and sizes of the two strings
+     that make up the visible portion of the buffer. */
 
   p1 = BEGV_ADDR;
   s1 = GPT_BYTE - BEGV_BYTE;
@@ -409,7 +407,6 @@ string_match_1 (Lisp_Object regexp, Lisp_Object string, Lisp_Object start,
 		   (NILP (Vinhibit_changing_match_data)
 		    ? &search_regs : NULL));
   immediate_quit = 0;
-  re_match_object = Qnil;       /* Stop protecting string from GC.  */
 
   /* Set last_thing_searched only when match data is changed.  */
   if (NILP (Vinhibit_changing_match_data))
@@ -480,7 +477,6 @@ fast_string_match_internal (Lisp_Object regexp, Lisp_Object string,
 		   SBYTES (string), 0,
 		   SBYTES (string), 0);
   immediate_quit = 0;
-  re_match_object = Qnil;       /* Stop protecting string from GC.  */
   return val;
 }
 
@@ -568,7 +564,6 @@ fast_looking_at (Lisp_Object regexp, ptrdiff_t pos, ptrdiff_t pos_byte,
   len = re_match_2 (buf, (char *) p1, s1, (char *) p2, s2,
 		    pos_byte, NULL, limit_byte);
   immediate_quit = 0;
-  re_match_object = Qnil;       /* Stop protecting string from GC.  */
 
   return len;
 }
@@ -1183,8 +1178,8 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 
   if (RE && !(trivial_regexp_p (string) && NILP (Vsearch_spaces_regexp)))
     {
-      unsigned char *base;
-      ptrdiff_t off1, off2, s1, s2;
+      unsigned char *p1, *p2;
+      ptrdiff_t s1, s2;
       struct re_pattern_buffer *bufp;
 
       bufp = compile_pattern (string,
@@ -1198,19 +1193,16 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 				   can take too long. */
       QUIT;			/* Do a pending quit right away,
 				   to avoid paradoxical behavior */
-      /* Get offsets and sizes of the two strings that make up the
-         visible portion of the buffer.  We compute offsets instead of
-         pointers because re_search_2 may call malloc and therefore
-         change the buffer text address.  */
+      /* Get pointers and sizes of the two strings
+	 that make up the visible portion of the buffer. */
 
-      base = current_buffer->text->beg;
-      off1 = BEGV_ADDR - base;
+      p1 = BEGV_ADDR;
       s1 = GPT_BYTE - BEGV_BYTE;
-      off2 = GAP_END_ADDR - base;
+      p2 = GAP_END_ADDR;
       s2 = ZV_BYTE - GPT_BYTE;
       if (s1 < 0)
 	{
-          off2 = off1;
+	  p2 = p1;
 	  s2 = ZV_BYTE - BEGV_BYTE;
 	  s1 = 0;
 	}
@@ -1225,16 +1217,12 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 	{
 	  ptrdiff_t val;
 
-          val = re_search_2 (bufp,
-                             (char*) (base + off1), s1,
-                             (char*) (base + off2), s2,
+	  val = re_search_2 (bufp, (char *) p1, s1, (char *) p2, s2,
 			     pos_byte - BEGV_BYTE, lim_byte - pos_byte,
 			     (NILP (Vinhibit_changing_match_data)
 			      ? &search_regs : &search_regs_1),
 			     /* Don't allow match past current point */
 			     pos_byte - BEGV_BYTE);
-	  /* Update 'base' due to possible relocation inside re_search_2.  */
-	  base = current_buffer->text->beg;
 	  if (val == -2)
 	    {
 	      matcher_overflow ();
@@ -1274,15 +1262,11 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
 	{
 	  ptrdiff_t val;
 
-          val = re_search_2 (bufp,
-                             (char*) (base + off1), s1,
-                             (char*) (base + off2), s2,
-                             pos_byte - BEGV_BYTE, lim_byte - pos_byte,
+	  val = re_search_2 (bufp, (char *) p1, s1, (char *) p2, s2,
+			     pos_byte - BEGV_BYTE, lim_byte - pos_byte,
 			     (NILP (Vinhibit_changing_match_data)
 			      ? &search_regs : &search_regs_1),
 			     lim_byte - BEGV_BYTE);
-	  /* Update 'base' due to possible relocation inside re_search_2.  */
-	  base = current_buffer->text->beg;
 	  if (val == -2)
 	    {
 	      matcher_overflow ();

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Sam Halliday]

Sorry Eli, this patch doesn't apply cleanly for me. I'm on the tip of
emacs-25 and I get failures in dired.c, regex.c and search.c that
aren't easy for me to resolve.

I'm happy to keep being guinea pig so you can fix this, but as a
workaround so that I can have a more stable Emacs to get my main work
done, is there a way for me to force the use of a less jumpy malloc
implementation? It sounds like whatever happened in ArchLinux has
meant that Emacs now prefers ralloc whereas previously it may not.

On 23 October 2016 at 20:32, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Sun, 23 Oct 2016 22:10:55 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: 24358@debbugs.gnu.org, npostavs@users.sourceforge.net
>>
>> The problem is to find the code which causes this.  Hmm...
>
> Sam, the patch below removes the changes done by Noam in an attempt to
> solve the original report of this bug.  Please apply it, and see if
> doing so allows you to complete your bootstrap without any errors.
>
> Noam, do you configure Emacs with --enable-check-lisp-object-type?
> If not, please do, because that's what Sam does; perhaps that is the
> reason why he gets the problem and you don't.
>
> One other difference is that Sam is running a bootstrap, so his
> bootstrap-emacs used to build ja-dic has the necessary Lisp files
> loaded as *.el files, not *.elc.  Therefore another thing to try
> (perhaps even before reconfiguring) is a full bootstrap, preferably in
> a separate clean directory, maybe you will see this problem then.
>
> Your systems seem to be similar, so I'd expect you both to see the
> same problems.
>
> Here's the patch to remove Noam's changes and Paul's follow-up
> changes:
>
> diff --git a/src/dired.c b/src/dired.c
> index 006f74c..dba575c 100644
> --- a/src/dired.c
> +++ b/src/dired.c
> @@ -259,11 +259,9 @@ directory_files_internal (Lisp_Object directory, Lisp_Object full,
>        QUIT;
>
>        bool wanted = (NILP (match)
> -                    || (re_match_object = name,
> -                         re_search (bufp, SSDATA (name), len, 0, len, 0) >= 0));
> +                    || re_search (bufp, SSDATA (name), len, 0, len, 0) >= 0);
>
>        immediate_quit = 0;
> -      re_match_object = Qnil;   /* Stop protecting name from GC.  */
>
>        if (wanted)
>         {
> diff --git a/src/regex.c b/src/regex.c
> index b12e95b..164eb46 100644
> --- a/src/regex.c
> +++ b/src/regex.c
> @@ -152,8 +152,6 @@
>
>  /* Converts the pointer to the char to BEG-based offset from the start.  */
>  # define PTR_TO_OFFSET(d) POS_AS_IN_BUFFER (POINTER_TO_OFFSET (d))
> -/* Strings are 0-indexed, buffers are 1-indexed; we pun on the boolean
> -   result to get the right base index.  */
>  # define POS_AS_IN_BUFFER(p) ((p) + (NILP (re_match_object) || BUFFERP (re_match_object)))
>
>  # define RE_MULTIBYTE_P(bufp) ((bufp)->multibyte)
> @@ -1438,62 +1436,11 @@ typedef struct
>  #define NEXT_FAILURE_HANDLE(h) fail_stack.stack[(h) - 3].integer
>  #define TOP_FAILURE_HANDLE() fail_stack.frame
>
> -#ifdef emacs
> -# define STR_BASE_PTR(obj)                      \
> -   (NILP (obj) ? current_buffer->text->beg      \
> -    : STRINGP (obj) ? SDATA (obj)               \
> -    : NULL)
> -#else
> -# define STR_BASE_PTR(obj) NULL
> -#endif
>
>  #define ENSURE_FAIL_STACK(space)                                       \
>  while (REMAINING_AVAIL_SLOTS <= space) {                               \
> -  re_char *orig_base = STR_BASE_PTR (re_match_object);                  \
> -  bool might_relocate = orig_base != NULL;                             \
> -  ptrdiff_t string1_off, end1_off, end_match_1_off;                     \
> -  ptrdiff_t string2_off, end2_off, end_match_2_off;                     \
> -  ptrdiff_t d_off, dend_off, dfail_off;                                 \
> -  if (might_relocate)                                                  \
> -    {                                                                   \
> -      if (string1)                                                      \
> -        {                                                               \
> -          string1_off = string1 - orig_base;                            \
> -          end1_off = end1 - orig_base;                                  \
> -          end_match_1_off = end_match_1 - orig_base;                    \
> -        }                                                               \
> -      if (string2)                                                      \
> -        {                                                               \
> -          string2_off = string2 - orig_base;                            \
> -          end2_off = end2 - orig_base;                                  \
> -          end_match_2_off = end_match_2 - orig_base;                    \
> -        }                                                               \
> -      d_off = d - orig_base;                                            \
> -      dend_off = dend - orig_base;                                      \
> -      dfail_off = dfail - orig_base;                                    \
> -    }                                                                   \
>    if (!GROW_FAIL_STACK (fail_stack))                                   \
>      return -2;                                                         \
> -  /* In Emacs, GROW_FAIL_STACK might relocate string pointers.  */     \
> -  if (might_relocate)                                                  \
> -    {                                                                   \
> -      re_char *new_base = STR_BASE_PTR (re_match_object);              \
> -      if (string1)                                                      \
> -        {                                                               \
> -          string1 = new_base + string1_off;                             \
> -          end1 = new_base + end1_off;                                   \
> -          end_match_1 = new_base + end_match_1_off;                     \
> -        }                                                               \
> -      if (string2)                                                      \
> -        {                                                               \
> -          string2 = new_base + string2_off;                             \
> -          end2 = new_base + end2_off;                                   \
> -          end_match_2 = new_base + end_match_2_off;                     \
> -        }                                                               \
> -      d = new_base + d_off;                                             \
> -      dend = new_base + dend_off;                                       \
> -      dfail = new_base + dfail_off;                                     \
> -    }                                                                   \
>    DEBUG_PRINT ("\n  Doubled stack; size now: %zd\n", (fail_stack).size);\
>    DEBUG_PRINT ("        slots available: %zd\n", REMAINING_AVAIL_SLOTS);\
>  }
> @@ -4380,10 +4327,6 @@ re_search_2 (struct re_pattern_buffer *bufp, const char *str1, size_t size1,
>    /* Loop through the string, looking for a place to start matching.  */
>    for (;;)
>      {
> -      ptrdiff_t offset1, offset2;
> -      re_char *orig_base;
> -      bool might_relocate;
> -
>        /* If the pattern is anchored,
>          skip quickly past places we cannot match.
>          We don't bother to treat startpos == 0 specially
> @@ -4500,17 +4443,6 @@ re_search_2 (struct re_pattern_buffer *bufp, const char *str1, size_t size1,
>           && !bufp->can_be_null)
>         return -1;
>
> -      /* re_match_2_internal may allocate, relocating the Lisp text
> -        object that we're searching.  */
> -      IF_LINT (offset2 = 0);  /* Work around GCC bug 78081.  */
> -      orig_base = STR_BASE_PTR (re_match_object);
> -      might_relocate = orig_base != NULL;
> -      if (might_relocate)
> -        {
> -          if (string1) offset1 = string1 - orig_base;
> -          if (string2) offset2 = string2 - orig_base;
> -        }
> -
>        val = re_match_2_internal (bufp, string1, size1, string2, size2,
>                                  startpos, regs, stop);
>
> @@ -4520,13 +4452,6 @@ re_search_2 (struct re_pattern_buffer *bufp, const char *str1, size_t size1,
>        if (val == -2)
>         return -2;
>
> -      if (might_relocate)
> -        {
> -         re_char *new_base = STR_BASE_PTR (re_match_object);
> -          if (string1) string1 = offset1 + new_base;
> -          if (string2) string2 = offset2 + new_base;
> -        }
> -
>      advance:
>        if (!range)
>         break;
> @@ -4962,8 +4887,8 @@ WEAK_ALIAS (__re_match, re_match)
>  #endif /* not emacs */
>
>  #ifdef emacs
> -/* In Emacs, this is the string or buffer in which we are matching.
> -   See the declaration in regex.h for details.  */
> +/* In Emacs, this is the string or buffer in which we
> +   are matching.  It is used for looking up syntax properties.  */
>  Lisp_Object re_match_object;
>  #endif
>
> diff --git a/src/search.c b/src/search.c
> index 5c04916..dc7e2d8 100644
> --- a/src/search.c
> +++ b/src/search.c
> @@ -287,10 +287,8 @@ looking_at_1 (Lisp_Object string, bool posix)
>    immediate_quit = 1;
>    QUIT;                        /* Do a pending quit right away, to avoid paradoxical behavior */
>
> -  /* Get pointers and sizes of the two strings that make up the
> -     visible portion of the buffer.  Note that we can use pointers
> -     here, unlike in search_buffer, because we only call re_match_2
> -     once, after which we never use the pointers again.  */
> +  /* Get pointers and sizes of the two strings
> +     that make up the visible portion of the buffer. */
>
>    p1 = BEGV_ADDR;
>    s1 = GPT_BYTE - BEGV_BYTE;
> @@ -409,7 +407,6 @@ string_match_1 (Lisp_Object regexp, Lisp_Object string, Lisp_Object start,
>                    (NILP (Vinhibit_changing_match_data)
>                     ? &search_regs : NULL));
>    immediate_quit = 0;
> -  re_match_object = Qnil;       /* Stop protecting string from GC.  */
>
>    /* Set last_thing_searched only when match data is changed.  */
>    if (NILP (Vinhibit_changing_match_data))
> @@ -480,7 +477,6 @@ fast_string_match_internal (Lisp_Object regexp, Lisp_Object string,
>                    SBYTES (string), 0,
>                    SBYTES (string), 0);
>    immediate_quit = 0;
> -  re_match_object = Qnil;       /* Stop protecting string from GC.  */
>    return val;
>  }
>
> @@ -568,7 +564,6 @@ fast_looking_at (Lisp_Object regexp, ptrdiff_t pos, ptrdiff_t pos_byte,
>    len = re_match_2 (buf, (char *) p1, s1, (char *) p2, s2,
>                     pos_byte, NULL, limit_byte);
>    immediate_quit = 0;
> -  re_match_object = Qnil;       /* Stop protecting string from GC.  */
>
>    return len;
>  }
> @@ -1183,8 +1178,8 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
>
>    if (RE && !(trivial_regexp_p (string) && NILP (Vsearch_spaces_regexp)))
>      {
> -      unsigned char *base;
> -      ptrdiff_t off1, off2, s1, s2;
> +      unsigned char *p1, *p2;
> +      ptrdiff_t s1, s2;
>        struct re_pattern_buffer *bufp;
>
>        bufp = compile_pattern (string,
> @@ -1198,19 +1193,16 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
>                                    can take too long. */
>        QUIT;                    /* Do a pending quit right away,
>                                    to avoid paradoxical behavior */
> -      /* Get offsets and sizes of the two strings that make up the
> -         visible portion of the buffer.  We compute offsets instead of
> -         pointers because re_search_2 may call malloc and therefore
> -         change the buffer text address.  */
> +      /* Get pointers and sizes of the two strings
> +        that make up the visible portion of the buffer. */
>
> -      base = current_buffer->text->beg;
> -      off1 = BEGV_ADDR - base;
> +      p1 = BEGV_ADDR;
>        s1 = GPT_BYTE - BEGV_BYTE;
> -      off2 = GAP_END_ADDR - base;
> +      p2 = GAP_END_ADDR;
>        s2 = ZV_BYTE - GPT_BYTE;
>        if (s1 < 0)
>         {
> -          off2 = off1;
> +         p2 = p1;
>           s2 = ZV_BYTE - BEGV_BYTE;
>           s1 = 0;
>         }
> @@ -1225,16 +1217,12 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
>         {
>           ptrdiff_t val;
>
> -          val = re_search_2 (bufp,
> -                             (char*) (base + off1), s1,
> -                             (char*) (base + off2), s2,
> +         val = re_search_2 (bufp, (char *) p1, s1, (char *) p2, s2,
>                              pos_byte - BEGV_BYTE, lim_byte - pos_byte,
>                              (NILP (Vinhibit_changing_match_data)
>                               ? &search_regs : &search_regs_1),
>                              /* Don't allow match past current point */
>                              pos_byte - BEGV_BYTE);
> -         /* Update 'base' due to possible relocation inside re_search_2.  */
> -         base = current_buffer->text->beg;
>           if (val == -2)
>             {
>               matcher_overflow ();
> @@ -1274,15 +1262,11 @@ search_buffer (Lisp_Object string, ptrdiff_t pos, ptrdiff_t pos_byte,
>         {
>           ptrdiff_t val;
>
> -          val = re_search_2 (bufp,
> -                             (char*) (base + off1), s1,
> -                             (char*) (base + off2), s2,
> -                             pos_byte - BEGV_BYTE, lim_byte - pos_byte,
> +         val = re_search_2 (bufp, (char *) p1, s1, (char *) p2, s2,
> +                            pos_byte - BEGV_BYTE, lim_byte - pos_byte,
>                              (NILP (Vinhibit_changing_match_data)
>                               ? &search_regs : &search_regs_1),
>                              lim_byte - BEGV_BYTE);
> -         /* Update 'base' due to possible relocation inside re_search_2.  */
> -         base = current_buffer->text->beg;
>           if (val == -2)
>             {
>               matcher_overflow ();

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: Sam Halliday <sam.halliday@gmail.com>
> Date: Sun, 23 Oct 2016 21:15:11 +0100
> Cc: 24358@debbugs.gnu.org, Noam Postavsky <npostavs@users.sourceforge.net>
> 
> Sorry Eli, this patch doesn't apply cleanly for me. I'm on the tip of
> emacs-25 and I get failures in dired.c, regex.c and search.c that
> aren't easy for me to resolve.

Strange, I prepared this with "git diff" on emacs-25 branch.

OK, can you reset dired.c, regex.c and search.c to their state as of
commit 5a26c9b0e1b0d9a2de35e0a8b0a803017e70def0 instead?  This is one
commit before Noam made his changes.

> I'm happy to keep being guinea pig so you can fix this, but as a
> workaround so that I can have a more stable Emacs to get my main work
> done, is there a way for me to force the use of a less jumpy malloc
> implementation?

Apart from downgrading to an older glibc version, I'm not aware of any
way.

> It sounds like whatever happened in ArchLinux has meant that Emacs
> now prefers ralloc whereas previously it may not.

Yes, I think so.

But we cannot ask users to not upgrade to newer glibc, so we must
solve these problems somehow.  If you can afford being a guinea pig
for a little bit longer, I'd appreciate that, as otherwise I don't
know how to proceed with this bug.  Other people are also seeing
similar problems, e.g., see

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24772#11

Thanks.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

Noam, there's something I don't understand here: why is regex.c using
malloc for allocating the failure stack?  AFAICS, REGEX_MALLOC is not
defined, so the only way I can explain this to myself is that we use
SAFE_ALLOCA, and that falls back to malloc because the failure stack
needs more than 16KB of memory.  Is that correct?

If so, then one other way of solving the original problem that doesn't
open the gates of the ralloc.c hell is to allow a larger amount of
stack allocations in regex.c before we fall back to malloc.  Since the
maximum size of the failure stack is limited by something like 800KB,
and the C stack limit is set in main.c to allow the failure stack be
allocated off the stack, why don't we make use of this fact?

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Noam Postavsky]

Eli Zaretskii <eliz@gnu.org> wrote:
> Noam, do you configure Emacs with --enable-check-lisp-object-type?
> If not, please do, because that's what Sam does; perhaps that is the
> reason why he gets the problem and you don't.
>
> One other difference is that Sam is running a bootstrap, so his
> bootstrap-emacs used to build ja-dic has the necessary Lisp files
> loaded as *.el files, not *.elc.  Therefore another thing to try
> (perhaps even before reconfiguring) is a full bootstrap, preferably in
> a separate clean directory, maybe you will see this problem then.

I will try both of these tomorrow.

Sam, if you want to post the docker thing somewhere I can give that a
try tomorrow as well, in case Eli's suggestions are not sufficient to
reproduce this.

> Noam, there's something I don't understand here: why is regex.c using
> malloc for allocating the failure stack?  AFAICS, REGEX_MALLOC is not
> defined, so the only way I can explain this to myself is that we use
> SAFE_ALLOCA, and that falls back to malloc because the failure stack
> needs more than 16KB of memory.  Is that correct?

Yes, this is correct, see also
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=24358#47

>
> If so, then one other way of solving the original problem that doesn't
> open the gates of the ralloc.c hell is to allow a larger amount of
> stack allocations in regex.c before we fall back to malloc.  Since the
> maximum size of the failure stack is limited by something like 800KB,
> and the C stack limit is set in main.c to allow the failure stack be
> allocated off the stack, why don't we make use of this fact?

It sounds reasonable in principle. This would mean changing MAX_ALLOCA?

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Sun, 23 Oct 2016 19:18:32 -0400
> Cc: Sam Halliday <sam.halliday@gmail.com>, 24358@debbugs.gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> wrote:
> > Noam, do you configure Emacs with --enable-check-lisp-object-type?
> > If not, please do, because that's what Sam does; perhaps that is the
> > reason why he gets the problem and you don't.
> >
> > One other difference is that Sam is running a bootstrap, so his
> > bootstrap-emacs used to build ja-dic has the necessary Lisp files
> > loaded as *.el files, not *.elc.  Therefore another thing to try
> > (perhaps even before reconfiguring) is a full bootstrap, preferably in
> > a separate clean directory, maybe you will see this problem then.
> 
> I will try both of these tomorrow.

Thanks.

Here are some other things to try, if you (or someone else) have time:

  . build regex.c with REGEX_MALLOC defined, and see whether your
    recent additions to cater to relocation are needed with that
    configuration
  . build with gmalloc.c, but without ralloc.c (I will try to come up
    with a recipe or a patch for that later today, if no one beats me
    to it)
  . surround the calls to regex.c functions in search.c and elsewhere
    with the calls to r_alloc_inhibit_buffer_relocation, again without
    your recent additions to regex.c (the problem with this is that it
    must be tested in long, real-life sessions, because in the past
    using that workaround caused very rare crashes in ralloc.c itself,
    which were supposed to be fixed, but were never tested thoroughly
    enough, AFAIK)

> > Noam, there's something I don't understand here: why is regex.c using
> > malloc for allocating the failure stack?  AFAICS, REGEX_MALLOC is not
> > defined, so the only way I can explain this to myself is that we use
> > SAFE_ALLOCA, and that falls back to malloc because the failure stack
> > needs more than 16KB of memory.  Is that correct?
> 
> Yes, this is correct, see also
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=24358#47
> 
> >
> > If so, then one other way of solving the original problem that doesn't
> > open the gates of the ralloc.c hell is to allow a larger amount of
> > stack allocations in regex.c before we fall back to malloc.  Since the
> > maximum size of the failure stack is limited by something like 800KB,
> > and the C stack limit is set in main.c to allow the failure stack be
> > allocated off the stack, why don't we make use of this fact?
> 
> It sounds reasonable in principle. This would mean changing MAX_ALLOCA?

Actually, it probably means using a different definition for
SAFE_ALLOCA in regex.c, or a separate macro.  I don't see the need to
increase stack usage elsewhere in Emacs.

Thanks.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Eli Zaretskii]

> Date: Mon, 24 Oct 2016 10:05:58 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: sam.halliday@gmail.com, 24358@debbugs.gnu.org
> 
>   . build with gmalloc.c, but without ralloc.c (I will try to come up
>     with a recipe or a patch for that later today, if no one beats me
>     to it)

This should work, I think:

  $ REL_ALLOC=no CFLAGS='<whatever>' ... ./configure ...

IOW, set REL_ALLOC=no before running the configure script, and leave
the rest of the settings and configure switches intact.  If this
works, you should see

  Should Emacs use a relocating allocator for buffers?   no

at the end of the configure run, and src/config.h should have

  /* Define REL_ALLOC if you want to use the relocating allocator for buffer
     space. */
  /* #undef REL_ALLOC */

gmalloc.c should still be compiled, but ralloc.c should not (delete
ralloc.o to be sure).

Thanks.

bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"

[Sam Halliday]

The failure occurs at 20% if I delete the line (which seems to have
moved to line 412).

Some more debugging, human readable first two frames:

(gdb) f 1
#1  0x00000000006351d9 in xsignal (error_symbol=..., data=...) at eval.c:1582
1582  Fsignal (error_symbol, data);
(gdb) p error_symbol
$8 = {
  i = 9600
}
(gdb) xpr
Lisp_Symbol
$9 = (struct Lisp_Symbol *) 0xdb0610 <lispsym+9600>
"args-out-of-range"
(gdb) p data
$10 = {
  i = 116316307
}
(gdb) xpr
Lisp_Cons
$11 = (struct Lisp_Cons *) 0x6eed890
{
  car = {
    i = 0x3cff3c4
  },
  u = {
    cdr = {
      i = 0x6eed8a3
    },
    chain = 0x6eed8a3
  }
}

(gdb) f 2
#2  0x00000000006352ae in xsignal3 (error_symbol=..., arg1=...,
arg2=..., arg3=...) at eval.c:1609
1609  xsignal (error_symbol, list3 (arg1, arg2, arg3));
(gdb) p error_symbol
$12 = {
  i = 9600
}
(gdb) xpr
Lisp_Symbol
$13 = (struct Lisp_Symbol *) 0xdb0610 <lispsym+9600>
"args-out-of-range"
(gdb) p arg1
$14 = {
  i = 63960004
}
(gdb) xpr
Lisp_String
$15 = (struct Lisp_String *) 0x3cff3c0
"七宝焼/"
(gdb) p arg2
$16 = {
  i = -18
}
(gdb) xpr
Lisp_Int1
$17 = -5
(gdb) p arg3
$18 = {
  i = 0
}
(gdb) xpr
Lisp_Symbol
$19 = (struct Lisp_Symbol *) 0xdae090 <lispsym>
"nil"




On 23 October 2016 at 18:19, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Sam Halliday <sam.halliday@gmail.com>
>> Date: Sun, 23 Oct 2016 16:58:07 +0100
>> Cc: npostavs@users.sourceforge.net, 24358@debbugs.gnu.org
>>
>> Processing OKURI-NASI entries ...
>> processed 10% ...
>> processed 20% ...
>> processed 30% ...
>> processed 40% ...
>>
>> Thread 1 "emacs" hit Breakpoint 2, Fsignal (error_symbol=...,
>> data=...) at eval.c:1476
>> 1476    = (NILP (error_symbol) ? Fcar (data) : error_symbol);
>> (gdb) xbacktrace
>> "substring" (0xffff6f00)
>> "string=" (0xffff70a0)
>> "if" (0xffff7230)
>> "while" (0xffff7400)
>> "catch" (0xffff7600)
>> "let" (0xffff7860)
>> "and" (0xffff79f0)
>> "let" (0xffff7c50)
>> "and" (0xffff7de0)
>> "or" (0xffff7f70)
>> "let" (0xffff81d0)
>> "skkdic-breakup-string" (0xffff8380)
>
> Thanks.
>
> Does it help to delete this line:
>
>   re_match_object = Qnil;       /* Stop protecting string from GC.  */
>
> in src/search.c:string_match_1 (should be line 404)?

When should ralloc.c be used? (WAS: bug#24358)

[npostavs]

Eli Zaretskii <eliz@gnu.org> writes:
>>     Thread 1 "emacs" hit Hardware watchpoint 4: current_buffer->text->beg
>> 
>>     Old value = (unsigned char *) 0x18351b8 ""
>>     New value = (unsigned char *) 0x188a1b8 ""
>>     r_alloc_sbrk (size=290816) at ralloc.c:818
>
> r_alloc_sbrk?  What OS is this?  We only use ralloc.c on a handful of
> them, as of Emacs 25.

Should ralloc.c be used on GNU/Linux systems that have GNU libc?

I found this in my config.log (I think it's related to ralloc use,
though I'm finding the configure code a bit confusing):

    configure:11440: checking whether malloc is Doug Lea style
    configure:11461: gcc -o conftest -O0 -g3 -march=native     conftest.c  >&5 
    conftest.c: In function 'main':
    conftest.c:107:6: error: '__malloc_initialize_hook' undeclared (first use in this function)
          __malloc_initialize_hook = hook;
          ^~~~~~~~~~~~~~~~~~~~~~~~

It seems that my malloc.h does not declare __malloc_initialize_hook,
even though 'man 3 malloc_hook' says

       #include <malloc.h>

       void *(*__malloc_hook)(size_t size, const void *caller);

       void *(*__realloc_hook)(void *ptr, size_t size, const void *caller);

       void *(*__memalign_hook)(size_t alignment, size_t size,
                                const void *caller);

       void (*__free_hook)(void *ptr, const void *caller);

       void (*__malloc_initialize_hook)(void);

       void (*__after_morecore_hook)(void);

Perhaps this is because the recommended way to set this hook (which is
different from what the configure test is using) doesn't require a
declaration?

       The variable __malloc_initialize_hook points at a function that is called once when the  mal‐
       loc  implementation  is initialized.  This is a weak variable, so it can be overridden in the
       application with a definition like the following:

           void (*__malloc_initialize_hook)(void) = my_init_hook;

Re: When should ralloc.c be used? (WAS: bug#24358)

[Paul Eggert]

npostavs@users.sourceforge.net wrote:
> Should ralloc.c be used on GNU/Linux systems that have GNU libc?

Yes, with bleeding-edge glibc, as __malloc_initialize_hook has been removed. 
Evidently your man page is out of sync with your glibc.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Fri, 21 Oct 2016 22:32:49 -0700
> 
> npostavs@users.sourceforge.net wrote:
> > Should ralloc.c be used on GNU/Linux systems that have GNU libc?
> 
> Yes, with bleeding-edge glibc, as __malloc_initialize_hook has been removed. 

If that's the case, shouldn't we switch such glibc systems to use mmap
instead?  It should be free of at least some of the problems in
ralloc.c, I think.

Alternatively, how about supporting an external Doug Lea malloc
library (assuming such a library exists and Emacs can be linked
against it)?

ralloc.c is generally "bad news", we've gone to non-trivial efforts
during the last years to reduce its usage to the minimum.  I always
thought that only MSDOS and perhaps a few *BSD systems still use it.
Having it creep back into GNU/Linux is really a bad regression, IMO.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Paul Eggert]

Eli Zaretskii wrote:
> Having it creep back into GNU/Linux is really a bad regression, IMO.

I don't like it either, but would rather work on redoing the build process so 
that we can use the native malloc on all hosts.

Re: When should ralloc.c be used?

[Stefan Monnier]

>> Having it creep back into GNU/Linux is really a bad regression, IMO.
> I don't like it either, but would rather work on redoing the build process
> so that we can use the native malloc on all hosts.

But that doesn't explain why we'd need to use ralloc in the mean time.


        Stefan

Re: When should ralloc.c be used?

[Paul Eggert]

Stefan Monnier wrote:
> that doesn't explain why we'd need to use ralloc in the mean time.

I suppose you're right that we don't need to; we could instead hack on Emacs to 
get it to work without ralloc on recent glibc. If someone wants to do that, 
great. I'd rather spend my own limited cycles on fixing the main problem, which 
is unexec.

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sat, 22 Oct 2016 19:37:36 -0700
> 
> Stefan Monnier wrote:
> > that doesn't explain why we'd need to use ralloc in the mean time.
> 
> I suppose you're right that we don't need to; we could instead hack on Emacs to 
> get it to work without ralloc on recent glibc.

How about using mmap in those cases?

> If someone wants to do that, great. I'd rather spend my own limited
> cycles on fixing the main problem, which is unexec.

I thought we agreed to get rid of unexec by loading a single .elc file
at startup of Emacs, and remove the distinction between temacs and
emacs altogether.  Is that what you'd like to work on?

Thanks.

Re: When should ralloc.c be used?

[Paul Eggert]

Eli Zaretskii wrote:

> How about using mmap in those cases?

I don't know, and would rather not spend time investigating.

> I thought we agreed to get rid of unexec by loading a single .elc file
> at startup of Emacs

Yes, if that performs well enough. We don't know yet whether it will. It's on my 
list of things to look into, but it's not trivial.

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sun, 23 Oct 2016 00:57:21 -0700
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > I thought we agreed to get rid of unexec by loading a single .elc file
> > at startup of Emacs
> 
> Yes, if that performs well enough. We don't know yet whether it will. It's on my 
> list of things to look into, but it's not trivial.

Can you share the concerns and the tests you'd like to be performed?
Perhaps others (myself included) could help with such testing.

Thanks.

Re: When should ralloc.c be used?

[Paul Eggert]

Eli Zaretskii wrote:
> Can you share the concerns and the tests you'd like to be performed?

I wrote something along those lines in Bug#23529; see the URL below. My main 
concern is startup time and energy. I don't have detailed benchmarks.

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23529#197

Re: When should ralloc.c be used?

[Eli Zaretskii]

Is it reasonable to require a version of glibc that still supports
__malloc_initialize_hook?  When people upgrade to a newer glibc, the
previous version is still left on the system, I presume (for programs
that need them which were built against those old versions)?

I took a look at our sources, and we have a lot of places where we
call malloc, directly or indirectly, while holding C pointers to data
of Lisp strings.  We also have several (maybe half a dozen) places
where the same happens with C pointers to buffer text.  Auditing all
of these and fixing them is a non-trivial job, so maybe we should try
to avoid the problems in the first place?  Is it a practical solution?

Re: When should ralloc.c be used?

[Stefan Monnier]

> Is it reasonable to require a version of glibc that still supports
> __malloc_initialize_hook?  When people upgrade to a newer glibc, the
> previous version is still left on the system, I presume (for programs
> that need them which were built against those old versions)?

Not necessarily, no.  E.g. it's not the case for fresh new installs.


        Stefan

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  emacs-devel@gnu.org
> Date: Sun, 23 Oct 2016 09:39:57 -0400
> 
> > Is it reasonable to require a version of glibc that still supports
> > __malloc_initialize_hook?  When people upgrade to a newer glibc, the
> > previous version is still left on the system, I presume (for programs
> > that need them which were built against those old versions)?
> 
> Not necessarily, no.  E.g. it's not the case for fresh new installs.

But they can downgrade, right?

Re: When should ralloc.c be used?

[Stefan Monnier]

>> > Is it reasonable to require a version of glibc that still supports
>> > __malloc_initialize_hook?  When people upgrade to a newer glibc, the
>> > previous version is still left on the system, I presume (for programs
>> > that need them which were built against those old versions)?
>> Not necessarily, no.  E.g. it's not the case for fresh new installs.
> But they can downgrade, right?

Depends on the details of the distribution, but while it's technically
probably possible, it's not necessarily simple for the end-user.


        Stefan

Re: When should ralloc.c be used?

[Paul Eggert]

Eli Zaretskii wrote:
> But they can downgrade, right?

No, as users don't necessarily have an older glibc to downgrade to. That train 
has already left the station. With my blessing, I might add.

Re: When should ralloc.c be used?

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sun, 23 Oct 2016 11:19:15 -0700
> 
> Eli Zaretskii wrote:
> > But they can downgrade, right?
> 
> No, as users don't necessarily have an older glibc to downgrade to. That train 
> has already left the station. With my blessing, I might add.

Then what are our choices to solve this for Emacs 25.2?  If GNU/Linux
starts using ralloc more and more, we will have crashes and data
corruption all over the place.  It's inconceivable to release 25.2 in
this state.

Any suggestions?

Re: When should ralloc.c be used?

[Stefan Monnier]

> Then what are our choices to solve this for Emacs 25.2?  If GNU/Linux
> starts using ralloc more and more, we will have crashes and data
> corruption all over the place.  It's inconceivable to release 25.2 in
> this state.

What's wrong with using gmalloc without ralloc and with mmap'd buffers?


        Stefan

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  emacs-devel@gnu.org
> Date: Sun, 23 Oct 2016 16:36:24 -0400
> 
> > Then what are our choices to solve this for Emacs 25.2?  If GNU/Linux
> > starts using ralloc more and more, we will have crashes and data
> > corruption all over the place.  It's inconceivable to release 25.2 in
> > this state.
> 
> What's wrong with using gmalloc without ralloc and with mmap'd buffers?

Nothing, if it works.  But someone should set up Emacs to do that, and
make sure the result builds, bootstraps, and works reliably,
i.e. doesn't have all the problems reported recently in this and
related bugs.

I don't have access to any platforms that are affected by this
(fencepost doesn't yet have such a new glibc).  I will do this myself
if no one else comes to help, but I really could use help from people
who work on platforms that are affected by this issue.  Noam and Sam
help, but we need more manpower and more expertise.

In any case, I asked what were our alternatives, because I'm not sure
we have a clear view of those.  Making decisions with just a peephole
view of the issues is never a good idea.  The best solution might not
be changing the configury to eliminate ralloc, it could be something
entirely different.

For example, I see in regex.c a set of special definitions for
REGEX_ALLOCATE_STACK and friends conditioned by this:

  #if defined REL_ALLOC && defined REGEX_MALLOC

These definitions call directly a few functions in ralloc.c, as
opposed to going via malloc.  Does anyone know what is this about?
Should we try building with REGEX_MALLOC on platforms that use
ralloc.c, and see whether the problems with regex searches triggered
by relocation go away?

Yet another idea is enlarge the stack space available to SAFE_ALLOCA
in regex.c, so that the failure stack is allocated off the C runtime
stack, thus side-stepping the relocation issues.

And maybe there are other possibilities.  We really need to come up
with all the possible ideas, try which ones work, and decide as
quickly as possible what is the best one.  This currently is the most
serious blocking issue on the way towards releasing Emacs 25.2 soon,
as we wanted to.

Re: When should ralloc.c be used?

[Eli Zaretskii]

> Date: Mon, 24 Oct 2016 09:54:12 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: eggert@cs.ucla.edu, emacs-devel@gnu.org
> 
> And maybe there are other possibilities.  We really need to come up
> with all the possible ideas, try which ones work, and decide as
> quickly as possible what is the best one.  This currently is the most
> serious blocking issue on the way towards releasing Emacs 25.2 soon,
> as we wanted to.

So I think the most promising alternatives at this point are:

  . Build with gmalloc but without ralloc.

    Would people who have ralloc.o in their src directory please
    reconfigure with REL_ALLOC=no, and see if the result works
    reliably in you're day-to-day work?  Please report the results
    here, and if you were hit by one of the related bugs (24358 and
    24764), please report also to the corresponding bug addresses.

  . Back-port the HYBRID_MALLOC changes from master.  Not sure if the
    patch is simple and safe enough, or whether the result is tested
    well enough to have that on emacs-25.

If one of these works, we should consider reverting the changes in
regex.c that attempt to handle relocation during regex.c calls.  We
should also consider removing ralloc.c from any of our builds, in the
hope that the platforms which we care about have a much better malloc
implementation than what was available 20 years ago.

Comments?

Re: When should ralloc.c be used?

[Paul Eggert]

Eli Zaretskii wrote:
> Then what are our choices to solve this for Emacs 25.2?

Sorry, I've lost context. What's "this"? Doesn't draft Emacs 25.2 work 
adequately on bleeding-edge glibc without our doing anything special? If not, 
what are the problems and how can we reproduce them?

I attempted to reproduce the problems, whatever they are, by building draft 
Emacs 25.2 with './configure emacs_cv_var_doug_lea_malloc=no' on x86-64 Ubuntu 
16.04.1 (I don't have easy access to 16.10 yet, but this does build and link 
gmalloc.o and ralloc.o). --enable-gcc-warnings did generate some warnings, which 
I just now fixed, but I didn't observe any runtime misbehaviors.

Re: When should ralloc.c be used?

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sun, 23 Oct 2016 21:59:52 -0700
> 
> Eli Zaretskii wrote:
> > Then what are our choices to solve this for Emacs 25.2?
> 
> Sorry, I've lost context. What's "this"?

The fact that Emacs 25.2 builds with ralloc.c on recent GNU/Linux
systems, which triggers random crashes, inability to build Emacs in
some cases, and other atrocities, such as corruption of buffer text.

> Doesn't draft Emacs 25.2 work adequately on bleeding-edge glibc
> without our doing anything special?

Not even close.

> If not, what are the problems and how can we reproduce them?

One problem is that relocation of buffer text and Lisp string data can
happen during regex searches, due to reallocation of the failure stack
to a size that exceeds MAX_ALLOCA.  Noam fixed that by some
non-trivial code in regex.c, but those changes seem to have uncovered
a problem that precludes bootstrapping Emacs 25.2, reported here:

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24358#123

which was independently reported for a different machine here:

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24772#11

I predict that more people will start hitting this as they upgrade to
newer glibc and start building Emacs with ralloc.c.

So we can't even build Emacs 25.2 reliably on "bleeding-edge"
GNU/Linux systems -- how's that for "working adequately"?

Then there's bug#24764, which sounds a lot like it's also related to
ralloc.c.  Because Michael Heerdegen talked about problems with
browsing Web pages, I looked at xml.c, and sure thing, it passes a C
pointer to buffer text to libxml2 functions which call malloc
internally.  I installed a ralloc-specific workaround for that, but
didn't yet hear from Michael if that was sufficient to solve his
frequent crashes in GC and corruptions of buffer text.

> I attempted to reproduce the problems, whatever they are, by building draft 
> Emacs 25.2 with './configure emacs_cv_var_doug_lea_malloc=no' on x86-64 Ubuntu 
> 16.04.1 (I don't have easy access to 16.10 yet, but this does build and link 
> gmalloc.o and ralloc.o). --enable-gcc-warnings did generate some warnings, which 
> I just now fixed, but I didn't observe any runtime misbehaviors.

The problems don't happen immediately, and the problem with bootstrap
(bug#24358) seems to be dependent on some factor we don't yet
understand: Noam cannot reproduce it, although his system is very
similar to the one where it does happen.  If you want an almost
immediate manifestation of the problem in a build with ralloc, remove
the calls to r_alloc_inhibit_buffer_relocation in xml.c, and browse
some Web pages, you will sooner or later hit an assertion violation in
parse_region.

More generally, I found a few more places in the sources where we hold
C pointers to buffer text around calls to functions that can call
malloc.  I'm not sure I found all of them, because the only way to
look for them I know of is not perfect.  As for the same situation
where we hold C pointers to Lisp string data where malloc can be
called, there are virtually dozens of them, the most frequent paradigm
is something like

  char *beg = SSDATA (lisp_string);
  char *end = beg + something;
  Lisp_Object new_string = make_unibyte_string (beg, end - beg);

(The catch here is that make_unibyte_string calls malloc internally,
which could relocate the data of the original lisp_string, and thus
invalidate the pointers 'beg' and 'end'.)

I don't remember well enough the internals of ralloc.c: perhaps it
doesn't relocate Lisp string data unless the string is long enough? or
at all?  So the problems with Lisp string might not be as grave as I
fear, but this should certainly be looked into before we dismiss all
those cases.

Bottom line: when GNU/Linux systems started using ralloc.c, they've
potentially exposed Emacs 25.2 to very serious instability, on the
platform that we consider by far the most important one.  We need to
move fast and thoroughly to investigate the possible solutions and
decide which one to use.

Re: When should ralloc.c be used?

[Andreas Schwab]

On Okt 24 2016, Eli Zaretskii <eliz@gnu.org> wrote:

> I don't remember well enough the internals of ralloc.c: perhaps it
> doesn't relocate Lisp string data unless the string is long enough? or
> at all?  So the problems with Lisp string might not be as grave as I
> fear, but this should certainly be looked into before we dismiss all
> those cases.

String data can be relocated even without ralloc, see
compact_small_strings.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  emacs-devel@gnu.org
> Date: Mon, 24 Oct 2016 10:29:35 +0200
> 
> On Okt 24 2016, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > I don't remember well enough the internals of ralloc.c: perhaps it
> > doesn't relocate Lisp string data unless the string is long enough? or
> > at all?  So the problems with Lisp string might not be as grave as I
> > fear, but this should certainly be looked into before we dismiss all
> > those cases.
> 
> String data can be relocated even without ralloc, see
> compact_small_strings.

Yes, but that's called only by GC, so not a problem in most (if not
all) the places I've seen that, and is not related to ralloc anyway.

Looking at ralloc.c and its callers, I think the only blocks of memory
it relocates on its own are those allocated with r_alloc or r_realloc,
which is only used for buffer text and (under REGEX_MALLOC) for
regex.c failure stack.  Which means compact_small_strings is the
_only_ place where string data is relocated, so just calls to malloc
cannot.

IOW, I confused GC and compact_small_strings with ralloc, when I
talked about string data, and our only problem is with pointers to
buffer text.

Am I missing something?

Thanks.

Re: When should ralloc.c be used?

[Paul Eggert]

On 10/24/2016 12:44 AM, Eli Zaretskii wrote:
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24358... Then there's 
> bug#24764 ...

These bugs seem to be fixed now (thanks to you!). As Andreas pointed 
out, the problems with ralloc.c are not as severe as initially feared, 
since they should be limited to pointers to buffer text and should not 
extend to pointers to Lisp strings.

As I understand it, although the ralloc.c approach worked for a long 
time, it fell out of favor on common platforms and so hasn't been 
debugged as thoroughly for the past several years. Unfortunately, recent 
changes to glibc have caused ralloc.c to be used again on common GNU 
platforms and this are shaking out longstanding bugs with the ralloc.c 
approach. This means people using bleeding-edge glibc are suffering 
problems similar to what people on now-unusual platforms must have had 
for some time.

Surely we can fix these ralloc.c-related bugs as they come up. That 
being said, they are a hassle for users and maintainers, and if dropping 
ralloc.c works and doesn't cause significant performance degradation it 
sounds like that would be a win.

Re: When should ralloc.c be used?

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 24 Oct 2016 09:21:50 -0700
> 
> On 10/24/2016 12:44 AM, Eli Zaretskii wrote:
> > https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24358... Then there's 
> > bug#24764 ...
> 
> These bugs seem to be fixed now (thanks to you!).

Some of them are fixed.  At least one more remains (I hope to fix it
soon).  What's more, we still don't know whether the changes in
regex.c by Noam are correct enough to solve the problems with
relocation of buffer text while we search the buffer, and we also
don't know yet whether the two bugs mentioned above are solved because
we didn't hear from their OPs.

Since these all are related to ralloc.c, the question is whether we
should get rid of using it on GNU/Linux, instead of chasing each of
these problems (which is anything but easy and might take time).

> As Andreas pointed out, the problems with ralloc.c are not as severe
> as initially feared, since they should be limited to pointers to
> buffer text and should not extend to pointers to Lisp strings.

Indeed, and that's a relief.

> As I understand it, although the ralloc.c approach worked for a long 
> time, it fell out of favor on common platforms and so hasn't been 
> debugged as thoroughly for the past several years. Unfortunately, recent 
> changes to glibc have caused ralloc.c to be used again on common GNU 
> platforms and this are shaking out longstanding bugs with the ralloc.c 
> approach. This means people using bleeding-edge glibc are suffering 
> problems similar to what people on now-unusual platforms must have had 
> for some time.

Yes, exactly.  And since most people at least here use Emacs on
GNU/Linux, the nasty problems due to ralloc.c are popping up much
faster and more frequently than they did when only *BSD and Windows
used ralloc.c.

> Surely we can fix these ralloc.c-related bugs as they come up. That 
> being said, they are a hassle for users and maintainers, and if dropping 
> ralloc.c works and doesn't cause significant performance degradation it 
> sounds like that would be a win.

Right, so I'd like your opinion and comments about the possible
solutions proposed so far:

  . Build with gmalloc but without ralloc.

  . Back-port the HYBRID_MALLOC changes from master.  Not sure if the
    patch is simple and safe enough, or whether the result is tested
    well enough to have that on emacs-25.

  . Build with gmalloc and use mmap for buffer text allocation.

Thanks.

Re: When should ralloc.c be used?

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 775 bytes --]

On 10/24/2016 09:39 AM, Eli Zaretskii wrote:
>
> Right, so I'd like your opinion and comments about the possible
> solutions proposed so far:
>
>    . Build with gmalloc but without ralloc.

This goes back to what we were doing, no?

>
>    . Back-port the HYBRID_MALLOC changes from master.  Not sure if the
>      patch is simple and safe enough, or whether the result is tested
>      well enough to have that on emacs-25.

This sounds riskier.

>    . Build with gmalloc and use mmap for buffer text allocation.

This also sounds riskier.

How about the attached patch for emacs-25? Basically, it says "use 
ralloc.c only if requested via './configure REL_ALLOC=yes'". I assume 
that this patch need not be ported to master, due to HYBRID_MALLOC. I 
haven't tested this.

[-- Attachment #2: ralloc.diff --]
[-- Type: text/x-patch, Size: 735 bytes --]

diff --git a/configure.ac b/configure.ac
index ae7dfe5..19b44bd 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2189,18 +2189,10 @@ if test "$doug_lea_malloc" = "yes" ; then
   AC_DEFINE(DOUG_LEA_MALLOC, 1,
     [Define to 1 if the system memory allocator is Doug Lea style,
      with malloc hooks and malloc_set_state.])
-
-  ## Use mmap directly for allocating larger buffers.
-  ## FIXME this comes from src/s/{gnu,gnu-linux}.h:
-  ## #ifdef DOUG_LEA_MALLOC; #undef REL_ALLOC; #endif
-  ## Does the AC_FUNC_MMAP test below make this check unnecessary?
-  case "$opsys" in
-    mingw32|gnu*) REL_ALLOC=no ;;
-  esac
 fi
 
 if test x"${REL_ALLOC}" = x; then
-  REL_ALLOC=${GNU_MALLOC}
+  REL_ALLOC=no
 fi
 
 use_mmap_for_buffers=no

Re: When should ralloc.c be used?

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 24 Oct 2016 09:54:24 -0700
> 
> >    . Build with gmalloc but without ralloc.
> 
> This goes back to what we were doing, no?

No, we were using the glibc malloc, AFAIK.  Or am I missing something?

> >    . Back-port the HYBRID_MALLOC changes from master.  Not sure if the
> >      patch is simple and safe enough, or whether the result is tested
> >      well enough to have that on emacs-25.
> 
> This sounds riskier.
> 
> >    . Build with gmalloc and use mmap for buffer text allocation.
> 
> This also sounds riskier.

I agree with your assessments.

> How about the attached patch for emacs-25? Basically, it says "use 
> ralloc.c only if requested via './configure REL_ALLOC=yes'".

LGTM.  Should we wait for people to build with REL_ALLOC=no manually,
to see if there are any problems, or should we push this right away?

> I assume that this patch need not be ported to master, due to
> HYBRID_MALLOC.

Yes, I think so.

Thanks.

Re: When should ralloc.c be used?

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 784 bytes --]

>>>    . Build with gmalloc but without ralloc.
>>
>> This goes back to what we were doing, no?
>
> No, we were using the glibc malloc, AFAIK.  Or am I missing something?

No you're right, I was sloppy.

>> How about the attached patch for emacs-25? Basically, it says "use
>> ralloc.c only if requested via './configure REL_ALLOC=yes'".
>
> LGTM.  Should we wait for people to build with REL_ALLOC=no manually,
> to see if there are any problems, or should we push this right away?

I doubt whether many more people will build with REL_ALLOC=no. So I think we 
should push it into emacs-25. Proposed patch attached. I have tested this on 
Fedora 24 x86-64 and Ubuntu 16.04 x86-64 with './configure 
emacs_cv_var_doug_lea_malloc=no' to simulate bleeding-edge glibc.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Default-REL_ALLOC-to-no.patch --]
[-- Type: text/x-diff; name="0001-Default-REL_ALLOC-to-no.patch", Size: 1732 bytes --]

From 561110345c48d48b4621d2e59487c2c17fcc988c Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Mon, 24 Oct 2016 23:11:32 -0700
Subject: [PATCH] Default REL_ALLOC to 'no'

This should make ralloc-related bugs less likely on GNU/Linux
systems with bleeding-edge glibc.  See the email thread containing:
http://lists.gnu.org/archive/html/emacs-devel/2016-10/msg00801.html
Do not merge to master.
* configure.ac (REL_ALLOC): Default to 'no' on all platforms, not
merely on platforms with Doug Lea malloc.  Although bleeding-edge
glibc no longer exports __malloc_initialize_hook and so longer
passes the configure-time test for Doug Lea malloc, ralloc tickles
longstanding bugs like Bug#24358 and Bug#24764 and Emacs is likely
to be more reliable without it.  This patch is not needed on
master, which uses hybrid malloc in this situation.
---
 configure.ac | 10 +---------
 1 file changed, 1 insertion(+), 9 deletions(-)

diff --git a/configure.ac b/configure.ac
index ae7dfe5..19b44bd 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2189,18 +2189,10 @@ if test "$doug_lea_malloc" = "yes" ; then
   AC_DEFINE(DOUG_LEA_MALLOC, 1,
     [Define to 1 if the system memory allocator is Doug Lea style,
      with malloc hooks and malloc_set_state.])
-
-  ## Use mmap directly for allocating larger buffers.
-  ## FIXME this comes from src/s/{gnu,gnu-linux}.h:
-  ## #ifdef DOUG_LEA_MALLOC; #undef REL_ALLOC; #endif
-  ## Does the AC_FUNC_MMAP test below make this check unnecessary?
-  case "$opsys" in
-    mingw32|gnu*) REL_ALLOC=no ;;
-  esac
 fi
 
 if test x"${REL_ALLOC}" = x; then
-  REL_ALLOC=${GNU_MALLOC}
+  REL_ALLOC=no
 fi
 
 use_mmap_for_buffers=no
-- 
2.7.4

Re: When should ralloc.c be used?

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 24 Oct 2016 23:23:45 -0700
> 
> > LGTM.  Should we wait for people to build with REL_ALLOC=no manually,
> > to see if there are any problems, or should we push this right away?
> 
> I doubt whether many more people will build with REL_ALLOC=no. So I think we 
> should push it into emacs-25. Proposed patch attached.

I agree.  Anyway, at least one person already tried this build and
reported that problems due to relocation are gone.

So please push to emacs-25.

Thanks.

Re: When should ralloc.c be used?

[Jérémie Courrèges-Anglas]

Eli Zaretskii <eliz@gnu.org> writes:

[...]

>> As I understand it, although the ralloc.c approach worked for a long 
>> time, it fell out of favor on common platforms and so hasn't been 
>> debugged as thoroughly for the past several years. Unfortunately, recent 
>> changes to glibc have caused ralloc.c to be used again on common GNU 
>> platforms and this are shaking out longstanding bugs with the ralloc.c 
>> approach. This means people using bleeding-edge glibc are suffering 
>> problems similar to what people on now-unusual platforms must have had 
>> for some time.
>
> Yes, exactly.  And since most people at least here use Emacs on
> GNU/Linux, the nasty problems due to ralloc.c are popping up much
> faster and more frequently than they did when only *BSD and Windows
> used ralloc.c.

I'm a bit surprised that such issues happen on recent glibc systems.
Emacs has been using ralloc on OpenBSD since years, and seems to be
pretty stable.  Granted, memory corruption bugs can depend on many
parameters, but still...

-- 
jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF  DDCC 0DFA 74AE 1524 E7EE

Re: When should ralloc.c be used?

[Jérémie Courrèges-Anglas]

Eli Zaretskii <eliz@gnu.org> writes:

[...]

>> As I understand it, although the ralloc.c approach worked for a long 
>> time, it fell out of favor on common platforms and so hasn't been 
>> debugged as thoroughly for the past several years. Unfortunately, recent 
>> changes to glibc have caused ralloc.c to be used again on common GNU 
>> platforms and this are shaking out longstanding bugs with the ralloc.c 
>> approach. This means people using bleeding-edge glibc are suffering 
>> problems similar to what people on now-unusual platforms must have had 
>> for some time.
>
> Yes, exactly.  And since most people at least here use Emacs on
> GNU/Linux, the nasty problems due to ralloc.c are popping up much
> faster and more frequently than they did when only *BSD and Windows
> used ralloc.c.

I'm a bit surprised that such issues happen on recent glibc systems.
Emacs has been using ralloc on OpenBSD since years, and seems to be
pretty stable.  Granted, memory corruption bugs can depend on many
parameters, but still...

-- 
jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF  DDCC 0DFA 74AE 1524 E7EE

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: jca@wxcvbn.org (Jérémie Courrèges-Anglas)
> Cc: Paul Eggert <eggert@cs.ucla.edu>, emacs-devel@gnu.org
> Date: Fri, 28 Oct 2016 08:19:48 +0200
> 
> > Yes, exactly.  And since most people at least here use Emacs on
> > GNU/Linux, the nasty problems due to ralloc.c are popping up much
> > faster and more frequently than they did when only *BSD and Windows
> > used ralloc.c.
> 
> I'm a bit surprised that such issues happen on recent glibc systems.
> Emacs has been using ralloc on OpenBSD since years, and seems to be
> pretty stable.  Granted, memory corruption bugs can depend on many
> parameters, but still...

I guess your usage patterns side-step the problematic code.  E.g., if
the resulting memory footprint is stable (i.e. never grows too much
too fast), ralloc will not need to relocate buffer text too
frequently, so you won't bump into these problems.  And some of those
problems appeared only recently: e.g., EWW, which triggers the problem
when it calls libxml2, is a 25.1 addition.

Re: When should ralloc.c be used?

[Andreas Schwab]

On Okt 23 2016, Eli Zaretskii <eliz@gnu.org> wrote:

> Is it reasonable to require a version of glibc that still supports
> __malloc_initialize_hook?  When people upgrade to a newer glibc, the
> previous version is still left on the system, I presume (for programs
> that need them which were built against those old versions)?

There will ever be only one glibc.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Sun, 23 Oct 2016 17:22:37 +0200
> Cc: Paul Eggert <eggert@cs.ucla.edu>, monnier@iro.umontreal.ca,
> 	emacs-devel@gnu.org
> 
> On Okt 23 2016, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > Is it reasonable to require a version of glibc that still supports
> > __malloc_initialize_hook?  When people upgrade to a newer glibc, the
> > previous version is still left on the system, I presume (for programs
> > that need them which were built against those old versions)?
> 
> There will ever be only one glibc.

Sorry, I don't understand what that means.  Can't it be that some
older program is linked against an older libc.so.N version?

Re: When should ralloc.c be used?

[Andreas Schwab]

On Okt 23 2016, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Date: Sun, 23 Oct 2016 17:22:37 +0200
>> Cc: Paul Eggert <eggert@cs.ucla.edu>, monnier@iro.umontreal.ca,
>> 	emacs-devel@gnu.org
>> 
>> On Okt 23 2016, Eli Zaretskii <eliz@gnu.org> wrote:
>> 
>> > Is it reasonable to require a version of glibc that still supports
>> > __malloc_initialize_hook?  When people upgrade to a newer glibc, the
>> > previous version is still left on the system, I presume (for programs
>> > that need them which were built against those old versions)?
>> 
>> There will ever be only one glibc.
>
> Sorry, I don't understand what that means.  Can't it be that some
> older program is linked against an older libc.so.N version?

There is no older version.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Sun, 23 Oct 2016 17:57:15 +0200
> Cc: eggert@cs.ucla.edu, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> >> > Is it reasonable to require a version of glibc that still supports
> >> > __malloc_initialize_hook?  When people upgrade to a newer glibc, the
> >> > previous version is still left on the system, I presume (for programs
> >> > that need them which were built against those old versions)?
> >> 
> >> There will ever be only one glibc.
> >
> > Sorry, I don't understand what that means.  Can't it be that some
> > older program is linked against an older libc.so.N version?
> 
> There is no older version.

That doesn't really help understanding the issues.  I guess it's above
my pay grade.

(When trying to solve such a grave problem, we need everybody's best
expertise and help, and you are one of the best experts on these
matters here.  Looks like I'm too naïve expecting such cooperation.)

Re: When should ralloc.c be used?

[Stefan Monnier]

> That doesn't really help understanding the issues.  I guess it's above
> my pay grade.

IIUC it's been libc.so.6 ever since distribution have moved from the
"old libc" to glibc-2.


        Stefan

Skipping unexec via a big .elc file (was: When should ralloc.c be used?)

[Stefan Monnier]

>> If someone wants to do that, great. I'd rather spend my own limited
>> cycles on fixing the main problem, which is unexec.
> I thought we agreed to get rid of unexec by loading a single .elc file
> at startup of Emacs, and remove the distinction between temacs and
> Emacs altogether.  Is that what you'd like to work on?

FWIW, I just did a quick experiment with the patch below which dumps the
state of Emacs's obarray after loadup.el into a big "dumped.elc" file.
Not sure if such an approach could work, but in any case I expect that
a working .elc file should likely be of comparable size.

The result is a .elc file of 3.3MB which seems reasonable.
When I try to load it, tho, I get:

    % time src/emacs -Q --batch -l dumped.elc -f kill-emacs
    src/emacs -Q --batch -l dumped.elc -f kill-emacs  3.50s user 0.00s system 99% cpu 3.506 total
    % 

And that's with a warm cache on "i3-4170 CPU @ 3.70GHz" (my first, and
still only, CPU that goes beyond 3GHz).

So even if there might be ways to speed this up, it doesn't look
too promising.


        Stefan


diff --git a/lisp/loadup.el b/lisp/loadup.el
index 5c16464..dddd71f 100644
--- a/lisp/loadup.el
+++ b/lisp/loadup.el
@@ -474,6 +474,65 @@
 						invocation-directory)
 			      (expand-file-name name invocation-directory)
 			      t)))
+      (message "Dumping into dumped.elc...preparing...")
+
+      ;; Dump the current state into a file so we can reload it!
+      (with-current-buffer (generate-new-buffer "dumped.elc")
+        (message "Dumping into dumped.elc...generating...")
+        (insert ";ELC\^W\^@\^@\^@\n;;; Compiled\n;;; in Emacs version " emacs-version "\n")
+        (let ((cmds '()))
+          (setcdr global-buffers-menu-map nil) ;; Get rid of buffer objects!
+          (mapatoms
+           (lambda (s)
+             (when (and (fboundp s)
+                        (not (subrp (symbol-function s)))
+                        ;; FIXME: We need these, but they contain
+                        ;; unprintable objects.
+                        (not (memq s '(rename-buffer))))
+               (push `(fset ',s ,(macroexp-quote (symbol-function s))) cmds))
+             (when (and (boundp s) (not (keywordp s))
+                        (not (memq s '(nil t
+                                       ;; I think we don't need these!
+                                       terminal-frame
+                                       ;; FIXME: We need these, but they contain
+                                       ;; unprintable objects.
+                                       advertised-signature-table
+                                       undo-auto--undoably-changed-buffers))))
+               ;; FIXME: Don't record in the load-history!
+               ;; FIXME: Handle varaliases!
+               (let ((v (symbol-value s)))
+                 (push `(defvar ,s
+                          ,(cond
+                            ((subrp v)
+                             `(symbol-function ',(intern (subr-name v))))
+                            ((and (markerp v) (null (marker-buffer v)))
+                             '(make-marker))
+                            ((and (overlayp v) (null (overlay-buffer v)))
+                             '(let ((ol (make-overlay (point-min) (point-min))))
+                                (delete-overlay ol)
+                                ol))
+                            (v (macroexp-quote v))))
+                       cmds)))
+             (when (symbol-plist s)
+               (push `(setplist ',s ',(symbol-plist s)) cmds))))
+          (message "Dumping into dumped.elc...printing...")
+          (let ((print-circle t)
+                (print-gensym t)
+                (print-quoted t)
+                (print-level nil)
+                (print-length nil)
+                (print-escape-newlines t))
+            (print `(progn . ,cmds) (current-buffer)))
+          (goto-char (point-min))
+          (while (re-search-forward " (\\(defvar\\|setplist\\|fset\\) " nil t)
+            (goto-char (match-beginning 0))
+            (delete-char 1) (insert "\n"))
+          (message "Dumping into dumped.elc...saving...")
+          (let ((coding-system-for-write 'emacs-internal))
+            (write-region (point-min) (point-max) (buffer-name)))
+          (message "Dumping into dumped.elc...done")
+          ))
+
       (kill-emacs)))
 
 ;; For machines with CANNOT_DUMP defined in config.h,

Re: Skipping unexec via a big .elc file (was: When should ralloc.c be used?)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 23 Oct 2016 12:44:33 -0400
> 
> >> If someone wants to do that, great. I'd rather spend my own limited
> >> cycles on fixing the main problem, which is unexec.
> > I thought we agreed to get rid of unexec by loading a single .elc file
> > at startup of Emacs, and remove the distinction between temacs and
> > Emacs altogether.  Is that what you'd like to work on?
> 
> FWIW, I just did a quick experiment with the patch below which dumps the
> state of Emacs's obarray after loadup.el into a big "dumped.elc" file.
> Not sure if such an approach could work, but in any case I expect that
> a working .elc file should likely be of comparable size.
> 
> The result is a .elc file of 3.3MB which seems reasonable.
> When I try to load it, tho, I get:
> 
>     % time src/emacs -Q --batch -l dumped.elc -f kill-emacs
>     src/emacs -Q --batch -l dumped.elc -f kill-emacs  3.50s user 0.00s system 99% cpu 3.506 total
>     % 
> 
> And that's with a warm cache on "i3-4170 CPU @ 3.70GHz" (my first, and
> still only, CPU that goes beyond 3GHz).
> 
> So even if there might be ways to speed this up, it doesn't look
> too promising.

That sounds strangely long, as I got less than 2 sec with all the
preloaded *.elc files concatenated to a single file, and that's before
I made pure-copy a no-op.

Another report was that "loadup" with pure-copy short-circuited took
less than 0.5 sec.  See

  https://lists.gnu.org/archive/html/emacs-devel/2016-01/msg01049.html

Was your Emacs an optimized build?

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

>   https://lists.gnu.org/archive/html/emacs-devel/2016-01/msg01049.html
> Was your Emacs an optimized build?

I tried it with my local build of master (not optimized) as well as with
the "emacs24" executable provided by Debian.  Both times were comparable.


        Stefan

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 23 Oct 2016 16:27:48 -0400
> 
> >   https://lists.gnu.org/archive/html/emacs-devel/2016-01/msg01049.html
> > Was your Emacs an optimized build?
> 
> I tried it with my local build of master (not optimized) as well as with
> the "emacs24" executable provided by Debian.  Both times were comparable.

An unoptimized Emacs runs about 3 times slower, so I cannot explain
your comparable results with both versions, it doesn't match any of my
experiences.

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

>> I tried it with my local build of master (not optimized) as well as with
>> the "emacs24" executable provided by Debian.  Both times were comparable.
> An unoptimized Emacs runs about 3 times slower,

In my experience it's much less drastic, unless you include
enable_checking and such in "unoptimized".

> so I cannot explain your comparable results with both versions, it
> doesn't match any of my experiences.

The way I explained it to myself is that the lread.c code is much
less affected (e.g. it should almost be unaffected by enable_checking).

BTW, have you tried my experiement on your side?


        Stefan

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Mon, 24 Oct 2016 08:47:49 -0400
> 
> > so I cannot explain your comparable results with both versions, it
> > doesn't match any of my experiences.
> 
> The way I explained it to myself is that the lread.c code is much
> less affected (e.g. it should almost be unaffected by enable_checking).

Reading Lisp involves a lot of CPU-intensive processing.

> BTW, have you tried my experiement on your side?

No.

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

>> The way I explained it to myself is that the lread.c code is much
>> less affected (e.g. it should almost be unaffected by enable_checking).
> Reading Lisp involves a lot of CPU-intensive processing.

Yes, but it's a different kind of code, so it may be
affected differently.
In any case, I have no concrete data to back up this intuition and
I don't believe it very strongly either.


        Stefan

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

> That sounds strangely long, as I got less than 2 sec with all the
> preloaded *.elc files concatenated to a single file, and that's before
> I made pure-copy a no-op.
> Another report was that "loadup" with pure-copy short-circuited took
> less than 0.5 sec.  See

Hmm... indeed, I got to 0.72s with his patch (on a different, slower
machine (a Thinkpad X201s, i.e. with a i7 CPU L620 @ 2.00GHz)).

If I re-add international/characters it goes up a bit to
0.96s, but still nowhere near the 3s I got on my big .elc file.
[ I wonder what makes loading my big file so slow.  ]

This said, there's still a factor 5-10 to get to "immediate", tho.


        Stefan

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Sun, 23 Oct 2016 21:07:47 -0400
> 
> > That sounds strangely long, as I got less than 2 sec with all the
> > preloaded *.elc files concatenated to a single file, and that's before
> > I made pure-copy a no-op.
> > Another report was that "loadup" with pure-copy short-circuited took
> > less than 0.5 sec.  See
> 
> Hmm... indeed, I got to 0.72s with his patch (on a different, slower
> machine (a Thinkpad X201s, i.e. with a i7 CPU L620 @ 2.00GHz)).
> 
> If I re-add international/characters it goes up a bit to
> 0.96s, but still nowhere near the 3s I got on my big .elc file.
> [ I wonder what makes loading my big file so slow.  ]
> 
> This said, there's still a factor 5-10 to get to "immediate", tho.

A small price to pay for the advantages, IMO.  The most important
advantage in my view is that the dumping/loading process becomes very
simple and understandable even by people with minimal knowledge of C
subtleties and Emacs internals, let alone development tools like the
assembler and the linker.  This would make future maintenance much
more robust and reliable, and also allow more contributors to work on
improving, speeding up, and extending the build process.  The
alternatives all require us to depend on a dwindling handful of
people, which is a huge disadvantage in the long run.

Re: Skipping unexec via a big .elc file

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> If I re-add international/characters it goes up a bit to
>> 0.96s, but still nowhere near the 3s I got on my big .elc file.
>> [ I wonder what makes loading my big file so slow.  ]
>> 
>> This said, there's still a factor 5-10 to get to "immediate", tho.
>
> A small price to pay for the advantages, IMO.

I think a one second startup time for "emacs -Q -nw" on a fast machine
sounds pretty horrific, myself.  Not all people live in Emacs, but
instead start and stop Emacs to do small edits to files.

It would also make using Emacs on slower, smaller mobile devices an
unsatisfying experience.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 24 Oct 2016 08:47:39 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> If I re-add international/characters it goes up a bit to
> >> 0.96s, but still nowhere near the 3s I got on my big .elc file.
> >> [ I wonder what makes loading my big file so slow.  ]
> >> 
> >> This said, there's still a factor 5-10 to get to "immediate", tho.
> >
> > A small price to pay for the advantages, IMO.
> 
> I think a one second startup time for "emacs -Q -nw" on a fast machine
> sounds pretty horrific, myself.

We are not talking about 1 sec, we are talking about less than half
that time, potentially even 1/4th of a second.

Re: Skipping unexec via a big .elc file

[Andreas Schwab]

On Okt 24 2016, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Date: Mon, 24 Oct 2016 08:47:39 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> If I re-add international/characters it goes up a bit to
>> >> 0.96s, but still nowhere near the 3s I got on my big .elc file.
>> >> [ I wonder what makes loading my big file so slow.  ]
>> >> 
>> >> This said, there's still a factor 5-10 to get to "immediate", tho.
>> >
>> > A small price to pay for the advantages, IMO.
>> 
>> I think a one second startup time for "emacs -Q -nw" on a fast machine
>> sounds pretty horrific, myself.
>
> We are not talking about 1 sec, we are talking about less than half
> that time, potentially even 1/4th of a second.

That's still a lot.

$ time emacs --batch --eval t
0.027user 0.011system 0m0.048selapsed 79.66%CPU

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  emacs-devel@gnu.org
> Date: Mon, 24 Oct 2016 10:24:26 +0200
> 
> > We are not talking about 1 sec, we are talking about less than half
> > that time, potentially even 1/4th of a second.
> 
> That's still a lot.
> 
> $ time emacs --batch --eval t
> 0.027user 0.011system 0m0.048selapsed 79.66%CPU

Then I guess you will have to continue using unexec, and when that
alternative disappears, switch to some other editor.

Re: Skipping unexec via a big .elc file

[Daniel Colascione]

On 10/24/2016 01:41 AM, Eli Zaretskii wrote:
>> From: Andreas Schwab <schwab@suse.de>
>> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  emacs-devel@gnu.org
>> Date: Mon, 24 Oct 2016 10:24:26 +0200
>>
>>> We are not talking about 1 sec, we are talking about less than half
>>> that time, potentially even 1/4th of a second.
>>
>> That's still a lot.
>>
>> $ time emacs --batch --eval t
>> 0.027user 0.011system 0m0.048selapsed 79.66%CPU
>
> Then I guess you will have to continue using unexec, and when that
> alternative disappears, switch to some other editor.
>

I have lots of scripts that run using emacs -Q --batch; many are invoked 
frequently in other scripts. Making each take 250ms instead of 27ms to 
run will greatly increase the overall runtime of the high-level 
operations. I don't see a need to regress performance here, since a 
custom malloc will perform at least as well as the last glibc malloc 
that supported unexec (since it could in principle be a literal copy of 
that code), and we found the performance of that malloc acceptable. I 
care _much_ more about runtime performance than I do about allocation 
throughput once started.

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> Cc: larsi@gnus.org, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 24 Oct 2016 02:47:03 -0700
> 
> >> $ time emacs --batch --eval t
> >> 0.027user 0.011system 0m0.048selapsed 79.66%CPU
> >
> > Then I guess you will have to continue using unexec, and when that
> > alternative disappears, switch to some other editor.
> >
> 
> I have lots of scripts that run using emacs -Q --batch; many are invoked 
> frequently in other scripts. Making each take 250ms instead of 27ms to 
> run will greatly increase the overall runtime of the high-level 
> operations.

Maybe --batch won't need to load all of the elc code, maybe we could
have a smaller batch.elc for that.  Or maybe what Ken just wrote will
bring the load time below 100 ms, who knows.

IOW, I think we are arguing prematurely about something whose
performance we don't really understand, haven't measured yet, and
haven't even written yet.  Doesn't sound like a good idea.

> I don't see a need to regress performance here, since a 
> custom malloc will perform at least as well as the last glibc malloc 
> that supported unexec (since it could in principle be a literal copy of 
> that code), and we found the performance of that malloc acceptable. I 
> care _much_ more about runtime performance than I do about allocation 
> throughput once started.

The desire to drop unexec is not just because of malloc, it's because
advances in compilers, linkers, and system security make maintenance
of unexec harder and harder.  For example, unexec is incompatible with
address sanitation and other similar security techniques.  It also
regularly breaks when some new section is invented by the linker.
Etc. etc.

Therefore, we already decided to move towards eliminating unexec, and
the only issue we should discuss is how to do that.  You are in fact
suggesting to overturn that decision, which I don't think people will
agree with.

Re: Skipping unexec via a big .elc file

[Daniel Colascione]

On 10/24/2016 03:00 AM, Eli Zaretskii wrote:
>> Cc: larsi@gnus.org, emacs-devel@gnu.org
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Mon, 24 Oct 2016 02:47:03 -0700
>>
>>>> $ time emacs --batch --eval t
>>>> 0.027user 0.011system 0m0.048selapsed 79.66%CPU
>>>
>>> Then I guess you will have to continue using unexec, and when that
>>> alternative disappears, switch to some other editor.
>>>
>>
>> I have lots of scripts that run using emacs -Q --batch; many are invoked
>> frequently in other scripts. Making each take 250ms instead of 27ms to
>> run will greatly increase the overall runtime of the high-level
>> operations.
>
> Maybe --batch won't need to load all of the elc code, maybe we could
> have a smaller batch.elc for that.  Or maybe what Ken just wrote will
> bring the load time below 100 ms, who knows.
>
> IOW, I think we are arguing prematurely about something whose
> performance we don't really understand, haven't measured yet, and
> haven't even written yet.  Doesn't sound like a good idea.
>
>> I don't see a need to regress performance here, since a
>> custom malloc will perform at least as well as the last glibc malloc
>> that supported unexec (since it could in principle be a literal copy of
>> that code), and we found the performance of that malloc acceptable. I
>> care _much_ more about runtime performance than I do about allocation
>> throughput once started.
>
> The desire to drop unexec is not just because of malloc, it's because
> advances in compilers, linkers, and system security make maintenance
> of unexec harder and harder.  For example, unexec is incompatible with
> address sanitation and other similar security techniques.  It also
> regularly breaks when some new section is invented by the linker.
> Etc. etc.
>
> Therefore, we already decided to move towards eliminating unexec, and
> the only issue we should discuss is how to do that.  You are in fact
> suggesting to overturn that decision, which I don't think people will
> agree with.

Sure. I'd like to have a PIE Emacs myself. We're talking about methods. 
I don't think the XEmacs-style "portable dumper" approach, with 
relocations, has been given adequate consideration.

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> Cc: schwab@suse.de, larsi@gnus.org, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 24 Oct 2016 03:03:37 -0700
> 
> I don't think the XEmacs-style "portable dumper" approach, with 
> relocations, has been given adequate consideration.

I think everyone agrees, which is why that approach is not being
considered.

But loading the "pre-loaded" *.elc files as quickly as possible is IMO
an attractive approach, because it's very simple and doesn't require
knowing too much about unrelated issues.

Re: Skipping unexec via a big .elc file

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 953 bytes --]

Eli Zaretskii <eliz@gnu.org> schrieb am Mo., 24. Okt. 2016 um 12:19 Uhr:

> > Cc: schwab@suse.de, larsi@gnus.org, emacs-devel@gnu.org
> > From: Daniel Colascione <dancol@dancol.org>
> > Date: Mon, 24 Oct 2016 03:03:37 -0700
> >
> > I don't think the XEmacs-style "portable dumper" approach, with
> > relocations, has been given adequate consideration.
>
> I think everyone agrees, which is why that approach is not being
> considered.
>
> But loading the "pre-loaded" *.elc files as quickly as possible is IMO
> an attractive approach, because it's very simple and doesn't require
> knowing too much about unrelated issues.
>
>
I agree, we should strife for simplicity first and performance later. I'd
suggest to use the pre-loaded .elc approach in master and work on a faster
(but still portable) replacement later, when the need arises. Switching to
a portable dumper now means we can cut out lots of code and workarounds,
which is a significant win.

[-- Attachment #2: Type: text/html, Size: 1862 bytes --]

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Mon, 24 Oct 2016 10:28:06 +0000
> Cc: schwab@suse.de, larsi@gnus.org, emacs-devel@gnu.org
> 
>  But loading the "pre-loaded" *.elc files as quickly as possible is IMO
>  an attractive approach, because it's very simple and doesn't require
>  knowing too much about unrelated issues.
> 
> I agree, we should strife for simplicity first and performance later. I'd suggest to use the pre-loaded .elc
> approach in master and work on a faster (but still portable) replacement later, when the need arises.

I agree: we should make it work right first, and speed it up later.
There's a lot of room for speed improvement, some of the ideas were
already voiced here.

If someone wants to work on this, I think some of the stuff that
should be done is this:

  . Implement a command that writes a given list of *.elc files into a
    single file.

  . Make the C code that today runs at dump time and records various
    build-related variables, such as source-directory and
    system-configuration-features, record the values in a Lisp file
    (eventually will be the same .elc file that is loaded at startup).

(I'm sure there are more items in that list, but I didn't think long
enough to come up with more.)

Volunteers are welcome.

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

>   . Implement a command that writes a given list of *.elc files into a
>     single file.

>   . Make the C code that today runs at dump time and records various
>     build-related variables, such as source-directory and
>     system-configuration-features, record the values in a Lisp file
>     (eventually will be the same .elc file that is loaded at startup).

BTW, my dumped.elc attempt was specifically trying to solve these
issues: by dumping the state of the obarray, we automatically get these
vars set like we want them.  It also solves other side-issues such as
making sure that `C-h f dolist RET' points to "subr.el" rather than to
"dumped.elc".


        Stefan

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 24 Oct 2016 09:52:54 -0400
> 
> >   . Implement a command that writes a given list of *.elc files into a
> >     single file.
> 
> >   . Make the C code that today runs at dump time and records various
> >     build-related variables, such as source-directory and
> >     system-configuration-features, record the values in a Lisp file
> >     (eventually will be the same .elc file that is loaded at startup).
> 
> BTW, my dumped.elc attempt was specifically trying to solve these
> issues: by dumping the state of the obarray, we automatically get these
> vars set like we want them.  It also solves other side-issues such as
> making sure that `C-h f dolist RET' points to "subr.el" rather than to
> "dumped.elc".

I consider tinkering with obarray's internals still too "advanced" to
prefer it to a simple generation of Lisp code that records values of a
few variables.  (And are you sure all of the information we record at
dump time is in obarray? I am not.)  The number of these variables is
not large, so finding them in the sources will not be hard.

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

> A small price to pay for the advantages, IMO.

I think some users will run away screaming if Emacs takes a whole second
to start up.

> The most important advantage in my view is that the dumping/loading
> process becomes very simple and understandable even by people with
> minimal knowledge of C subtleties and Emacs internals,

Yes, the benefits are clear, but the cost is pretty steep.

I think we could live with a 0.2s startup time, but that's already
a pretty high cost:
- 0.2s feels sluggish when you expect "immediate".
- byte-compilation has historically moved from "do it in a single
  session", to "start a separate Emacs session for each file" for good
  reasons.  A 0.2s startup time imposes either a much slower
  byte-compilation, or will compel us to go back to "do it all in
  a single session".

> This would make future maintenance much more robust and reliable, and
> also allow more contributors to work on improving, speeding up, and
> extending the build process.  The alternatives all require us to
> depend on a dwindling handful of people, which is a huge disadvantage
> in the long run.

Maybe there's indeed a lot of speed up still waiting there, and by
reducing loading time of .elc files (and/or allowing more laziness there)
we could bring down the 0.96s to 0.2s *and* speed up other uses at the
same time.


        Stefan

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Mon, 24 Oct 2016 09:04:29 -0400
> 
> > A small price to pay for the advantages, IMO.
> 
> I think some users will run away screaming if Emacs takes a whole second
> to start up.

It depends.  If those users, like me, have hundreds of buffers in
their sessions, and use desktop.el to recreate their sessions, they
already wait a few seconds for that.

And I don't expect the result to be 1 sec, that's is a rounded up
value that is already higher than what I saw.

> > The most important advantage in my view is that the dumping/loading
> > process becomes very simple and understandable even by people with
> > minimal knowledge of C subtleties and Emacs internals,
> 
> Yes, the benefits are clear, but the cost is pretty steep.

We will have to speed this up, of course.  You didn't expect tossing
unexec to be an easy job, did you?

> I think we could live with a 0.2s startup time, but that's already
> a pretty high cost:
> - 0.2s feels sluggish when you expect "immediate".
> - byte-compilation has historically moved from "do it in a single
>   session", to "start a separate Emacs session for each file" for good
>   reasons.  A 0.2s startup time imposes either a much slower
>   byte-compilation, or will compel us to go back to "do it all in
>   a single session".

I think you forget parallelism.  We build Emacs with several
compilations running in parallel for a long time.  And byte-compiling
a typical file already takes more than 0.2 sec, sometimes (often?)
significantly more, so I don't see a catastrophe yet.

> > This would make future maintenance much more robust and reliable, and
> > also allow more contributors to work on improving, speeding up, and
> > extending the build process.  The alternatives all require us to
> > depend on a dwindling handful of people, which is a huge disadvantage
> > in the long run.
> 
> Maybe there's indeed a lot of speed up still waiting there, and by
> reducing loading time of .elc files (and/or allowing more laziness there)
> we could bring down the 0.96s to 0.2s *and* speed up other uses at the
> same time.

That's my hope, yes.  E.g., maybe reading the startup.elc file could
run in another thread?

In any case, I don't think it's right to throw out this idea without
trying very hard to make it work, because the benefits are so clear.

Re: Skipping unexec via a big .elc file

[Daniel Colascione]

On 10/24/2016 06:35 AM, Eli Zaretskii wrote:
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: emacs-devel@gnu.org
>> Date: Mon, 24 Oct 2016 09:04:29 -0400
>>
>>> A small price to pay for the advantages, IMO.
>>
>> I think some users will run away screaming if Emacs takes a whole second
>> to start up.
>
> It depends.  If those users, like me, have hundreds of buffers in
> their sessions, and use desktop.el to recreate their sessions, they
> already wait a few seconds for that.
>
> And I don't expect the result to be 1 sec, that's is a rounded up
> value that is already higher than what I saw.
>
>>> The most important advantage in my view is that the dumping/loading
>>> process becomes very simple and understandable even by people with
>>> minimal knowledge of C subtleties and Emacs internals,
>>
>> Yes, the benefits are clear, but the cost is pretty steep.
>
> We will have to speed this up, of course.  You didn't expect tossing
> unexec to be an easy job, did you?
>
>> I think we could live with a 0.2s startup time, but that's already
>> a pretty high cost:
>> - 0.2s feels sluggish when you expect "immediate".
>> - byte-compilation has historically moved from "do it in a single
>>   session", to "start a separate Emacs session for each file" for good
>>   reasons.  A 0.2s startup time imposes either a much slower
>>   byte-compilation, or will compel us to go back to "do it all in
>>   a single session".
>
> I think you forget parallelism.  We build Emacs with several
> compilations running in parallel for a long time.  And byte-compiling
> a typical file already takes more than 0.2 sec, sometimes (often?)
> significantly more, so I don't see a catastrophe yet.
>
>>> This would make future maintenance much more robust and reliable, and
>>> also allow more contributors to work on improving, speeding up, and
>>> extending the build process.  The alternatives all require us to
>>> depend on a dwindling handful of people, which is a huge disadvantage
>>> in the long run.
>>
>> Maybe there's indeed a lot of speed up still waiting there, and by
>> reducing loading time of .elc files (and/or allowing more laziness there)
>> we could bring down the 0.96s to 0.2s *and* speed up other uses at the
>> same time.
>
> That's my hope, yes.  E.g., maybe reading the startup.elc file could
> run in another thread?
>
> In any case, I don't think it's right to throw out this idea without
> trying very hard to make it work, because the benefits are so clear.

I'm worried that it'll be deemed to "work" at a level of performance 
much worse than what we have today. My preference would be to keep 
hammering on this approach and others until we find something with only 
minimal performance regressions. I don't see the unexec maintenance 
situation being desperate enough that we need to accept a big 
performance loss.

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 24 Oct 2016 07:45:17 -0700
> 
> > In any case, I don't think it's right to throw out this idea without
> > trying very hard to make it work, because the benefits are so clear.
> 
> I'm worried that it'll be deemed to "work" at a level of performance 
> much worse than what we have today.

Why would you worry that it'll be accepted then more easily than it's
accepted now?  The same arguments will be voiced in the future if the
solution's performance turns out to be insufficient.

> I don't see the unexec maintenance situation being desperate enough
> that we need to accept a big performance loss.

I very much disagree with this: the unexec maintenance situation is
actually so fragile that it could break at any moment, in the sense
that we could very easily get into having no people on board who know
enough about unexec to solve the next problem that will break it.  The
number of people who do know gets smaller and smaller with each year.
That is not healthy at all for the future of the project.

Re: Skipping unexec via a big .elc file

[Daniel Colascione]

On 10/24/2016 08:58 AM, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Mon, 24 Oct 2016 07:45:17 -0700
>>
>>> In any case, I don't think it's right to throw out this idea without
>>> trying very hard to make it work, because the benefits are so clear.
>>
>> I'm worried that it'll be deemed to "work" at a level of performance
>> much worse than what we have today.
>
> Why would you worry that it'll be accepted then more easily than it's
> accepted now?  The same arguments will be voiced in the future if the
> solution's performance turns out to be insufficient.
>
>> I don't see the unexec maintenance situation being desperate enough
>> that we need to accept a big performance loss.
>
> I very much disagree with this: the unexec maintenance situation is
> actually so fragile that it could break at any moment, in the sense
> that we could very easily get into having no people on board who know
> enough about unexec to solve the next problem that will break it.  The
> number of people who do know gets smaller and smaller with each year.
> That is not healthy at all for the future of the project.

In both this discussion and the one about insdel, you've expressed the 
sentiment that we need to optimize for a world in which very few people 
have time to maintain Emacs internals. I have a more optimistic view: 
people are generally good at figuring things out, and if learning about 
unexec or other esoteric facilities is that prevents a developer from 
porting Emacs to a new platform or fixing an important bug, that 
developer will put time into learning about these mechanisms.

That is, we *could* get into a situation where "no people on board [] 
know enough about unexec to solve the next problem", but that situation 
will resolve itself when people learn about unexec.

Re: Skipping unexec via a big .elc file

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 703 bytes --]

Daniel Colascione <dancol@dancol.org> schrieb am Mo., 24. Okt. 2016 um
18:35 Uhr:

> That is, we *could* get into a situation where "no people on board []
> know enough about unexec to solve the next problem"


I'd argue that we are already in this situation.  For example, nobody knows
how to make unexec work with ASLR or PIE; when I tried fuzzing Emacs with
AFL, the dumped binary would simply crash; the dumped binary is not
reproducible (i.e. bit-by-bit identical after every build); and I think
dumping also doesn't work with ASan. The fraction of situation where unexec
doesn't work any more gets larger and larger. If we had people who could
solve these problems, it should get smaller instead.

[-- Attachment #2: Type: text/html, Size: 1013 bytes --]

Re: Skipping unexec via a big .elc file

[Daniel Colascione]

Philipp Stephani <p.stephani2@gmail.com> writes:

> Daniel Colascione <dancol@dancol.org> schrieb am Mo., 24. Okt. 2016 um 18:35 Uhr:
>
>  That is, we *could* get into a situation where "no people on board []
>  know enough about unexec to solve the next problem"
>
> I'd argue that we are already in this situation.  For example, nobody
> knows how to make unexec work with ASLR or PIE; when I tried fuzzing
> Emacs with AFL, the dumped binary would simply crash; the dumped
> binary is not reproducible (i.e. bit-by-bit identical after every
> build); and I think dumping also doesn't work with ASan. The fraction
> of situation where unexec doesn't work any more gets larger and
> larger. If we had people who could solve these problems, it should get
> smaller instead.

It's not a matter of "not knowing" how to make unexec work with PIE and
PIC code generally --- the problem is that the naive approach currently
used for serializing program state depends on the process address state
being reproducible: we don't specially mark pointers in the saved image,
so we can't relocate them. There have been numerous discussions on
emacs-devel about relocation schemes, with proposals ranging from just
making elc faster to translating elisp to C.

Everyone who's seriously thought about the unexec problem _understands_
the issue. unexec isn't black magic. Getting rid of the current scheme
is a matter of finding the right relocation scheme (which for all I know
might as well be "make elc better") and finding the time to
implement it.

My preferred approach is the portable dumper one: basically what we're
doing today, except that instead of just blindly copying the data
segment and heap to a new emacs binary, we'll write this information to
a separate file, stored in a portable format, a file that we'll keep
alongside the Emacs binary.  We'll store in this file metadata about
where the pointers are. (There are two kinds of pointers in this file:
pointers to other parts of the file and pointers to the Emacs binary.)

At startup, we'll load the dump file and walk the relocations, fixing up
all the embedded addresses to account for the new process's different
address space.  There's no binary other than the one that the compiler
generates; this data file is just data, so ASLR, ASAN, and other clever
things should work fine. (Some people have proposed asking the system
dynamic linker to do the relocating, but I'd prefer to do it ourselves,
in a portable way.)

We can't save all of the Emacs data segment this way, but we can
relocate and restore anything that's marked with staticpro. The overall
experience should be very similar to what we have today.

Additionally, the purespace concept remains useful: if we take pure
storage and put it in its own region of the dump file, we don't need to
take copy-on-write faults for data that cannot contain pointers.

Speaking of COW faults: a refinement of this scheme is to do the
relocations lazily, in a SIGSEGV handler.  (Map the dump file PROT_NONE
so any access traps.)  In the SIGSEGV handler, we can relocate just the
page we faulted, then continue. This way, we don't need to slurp in the
entire dump file from disk just to start emacs -Q -batch: we can
demand-page!

Whether this refinement is worth the trouble is something only
experimentation can tell, but it's an option if we need it.  With this
refinement, the portable dumping approach should be safe, semantically
familiar to unexec, ASLR-compatible, _and_ very nearly as fast as what
we have today.

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Mon, 24 Oct 2016 12:47:56 -0700
> 
> > I'd argue that we are already in this situation.  For example, nobody
> > knows how to make unexec work with ASLR or PIE; when I tried fuzzing
> > Emacs with AFL, the dumped binary would simply crash; the dumped
> > binary is not reproducible (i.e. bit-by-bit identical after every
> > build); and I think dumping also doesn't work with ASan. The fraction
> > of situation where unexec doesn't work any more gets larger and
> > larger. If we had people who could solve these problems, it should get
> > smaller instead.
> 
> Everyone who's seriously thought about the unexec problem _understands_
> the issue.

The important point is that the number of people here who can claim
such understanding, enough so to fix the issues, is diminishingly
small, and gets smaller every year.

> My preferred approach is the portable dumper one: basically what we're
> doing today, except that instead of just blindly copying the data
> segment and heap to a new emacs binary, we'll write this information to
> a separate file, stored in a portable format, a file that we'll keep
> alongside the Emacs binary.  We'll store in this file metadata about
> where the pointers are. (There are two kinds of pointers in this file:
> pointers to other parts of the file and pointers to the Emacs binary.)
> 
> At startup, we'll load the dump file and walk the relocations, fixing up
> all the embedded addresses to account for the new process's different
> address space.

Why do you think this will have better performance that reading a
single .elc file at startup?  It's still mainly file I/O and
processing of the file's contents, just like with byte-compiled files.

If we have no reason to believe this portable dumper will be
significantly faster, we should IMO investigate the .elc method first,
because it's so much simpler, both in its implementation and in future
maintenance.  E.g., adding a new kind of Lisp object to Emacs would
require corresponding changes in the dumper.

> We can't save all of the Emacs data segment this way, but we can
> relocate and restore anything that's marked with staticpro. The overall
> experience should be very similar to what we have today.
> [...]
> Speaking of COW faults: a refinement of this scheme is to do the
> relocations lazily, in a SIGSEGV handler.  (Map the dump file PROT_NONE
> so any access traps.)  In the SIGSEGV handler, we can relocate just the
> page we faulted, then continue. This way, we don't need to slurp in the
> entire dump file from disk just to start emacs -Q -batch: we can
> demand-page!

Demand paging in an application, and an application such as Emacs on
top of that, makes little sense to me.  This is the OS business, not
ours.  Using mmap as a fast way to read a file, yes, that's done in
many applications.  But please lets leave demand paging out of our
scope.

IMO the less we mess with low-level techniques that no other
applications use the better, both because we have very few people who
can do that and because doing so runs higher risk of becoming broken
by future developments in the platforms we deem important.  The
long-term tendency in Emacs development should be to move away from
such techniques, not to acquire more of them.

Re: Skipping unexec via a big .elc file

[Daniel Colascione]

On 10/25/2016 08:59 AM, Eli Zaretskii wrote:
>> From: Daniel Colascione <dancol@dancol.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Mon, 24 Oct 2016 12:47:56 -0700
>>
>>> I'd argue that we are already in this situation.  For example, nobody
>>> knows how to make unexec work with ASLR or PIE; when I tried fuzzing
>>> Emacs with AFL, the dumped binary would simply crash; the dumped
>>> binary is not reproducible (i.e. bit-by-bit identical after every
>>> build); and I think dumping also doesn't work with ASan. The fraction
>>> of situation where unexec doesn't work any more gets larger and
>>> larger. If we had people who could solve these problems, it should get
>>> smaller instead.
>>
>> Everyone who's seriously thought about the unexec problem _understands_
>> the issue.
>
> The important point is that the number of people here who can claim
> such understanding, enough so to fix the issues, is diminishingly
> small, and gets smaller every year.

There's no demand for more yet. There isn't a catastrophe --- just low 
demand for core-change expertise. There used* to be a lot more (at least 
per-capita) stonemasons in historical societies than in today's society. 
That doesn't mean we've forgotten how to cut stones, and if there were a 
sudden need to do it, more stonemasons would magically appear.

>> My preferred approach is the portable dumper one: basically what we're
>> doing today, except that instead of just blindly copying the data
>> segment and heap to a new emacs binary, we'll write this information to
>> a separate file, stored in a portable format, a file that we'll keep
>> alongside the Emacs binary.  We'll store in this file metadata about
>> where the pointers are. (There are two kinds of pointers in this file:
>> pointers to other parts of the file and pointers to the Emacs binary.)
>>
>> At startup, we'll load the dump file and walk the relocations, fixing up
>> all the embedded addresses to account for the new process's different
>> address space.
>
> Why do you think this will have better performance that reading a
> single .elc file at startup?  It's still mainly file I/O and
> processing of the file's contents, just like with byte-compiled files.

Because a portable dumper can do less, on both file I/O and processing 
of the file's contents. There's no lisp evaluation, no slurping a whole 
file into memory. Having to read all of Emacs into memory on startup is 
a burden even on a fast, modern machine like mine.

~/edev/trunk/src
$ sync && sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

~/edev/trunk/src
$ time pv < emacs >/dev/null
48.6MiB 0:00:00 [ 455MiB/s] 
[=========================================================>] 100% 


real	0m0.116s
user	0m0.000s
sys	0m0.016s

That's pretty fast, but it's not free. Not having to do this much IO on 
startup in the first place would be even better.

> If we have no reason to believe this portable dumper will be
> significantly faster, we should IMO investigate the .elc method first,
> because it's so much simpler, both in its implementation and in future
> maintenance.  E.g., adding a new kind of Lisp object to Emacs would
> require corresponding changes in the dumper.

Adding a new kind of lisp object requires changes throughout core 
anyway. At the very least, you need to teach GC where your new object 
keeps its pointers, and that's exactly the knowledge that the dumper 
would need.

>> We can't save all of the Emacs data segment this way, but we can
>> relocate and restore anything that's marked with staticpro. The overall
>> experience should be very similar to what we have today.
>> [...]
>> Speaking of COW faults: a refinement of this scheme is to do the
>> relocations lazily, in a SIGSEGV handler.  (Map the dump file PROT_NONE
>> so any access traps.)  In the SIGSEGV handler, we can relocate just the
>> page we faulted, then continue. This way, we don't need to slurp in the
>> entire dump file from disk just to start emacs -Q -batch: we can
>> demand-page!
>
> Demand paging in an application, and an application such as Emacs on
> top of that, makes little sense to me.

Why? It's conceptually no different from autoload. There is no technique 
in computer science so rarefied that it's only good in ring zero.

> This is the OS business, not
> ours.  Using mmap as a fast way to read a file, yes, that's done in
> many applications.  But please lets leave demand paging out of our
> scope.

Emacs isn't just an application. It's a Lisp virtual machine, and 
employing the optimization techniques used in other virtual machines can 
be important wins.

(FWIW, mmap isn't a particularly fast way of doing bulk file reads. 
That's why GNU grep removed its mmap support.)

> IMO the less we mess with low-level techniques that no other
> applications use the better, both because we have very few people who
> can do that and because doing so runs higher risk of becoming broken
> by future developments in the platforms we deem important.  The
> long-term tendency in Emacs development should be to move away from
> such techniques, not to acquire more of them.

I'm for anything that delivers meaningful performance advantages.

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> Cc: p.stephani2@gmail.com, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Tue, 25 Oct 2016 09:14:55 -0700
> 
> >> Everyone who's seriously thought about the unexec problem _understands_
> >> the issue.
> >
> > The important point is that the number of people here who can claim
> > such understanding, enough so to fix the issues, is diminishingly
> > small, and gets smaller every year.
> 
> There's no demand for more yet.

Not true.  Demand for this level of expertise is continuous in Emacs,
and never dwindles, not for the last 25 years that I'm involved.

> There used* to be a lot more (at least 
> per-capita) stonemasons in historical societies than in today's society. 
> That doesn't mean we've forgotten how to cut stones, and if there were a 
> sudden need to do it, more stonemasons would magically appear.

I think your optimism is misplaced.  I'm old enough to have seen
several proficiencies go extinct due to new technology that made them
irrelevant.  When demand for those forgotten proficiencies came up,
people invariably run to the few still around who know how to do that,
they don't learn that themselves (and don't even know how).

> > Why do you think this will have better performance that reading a
> > single .elc file at startup?  It's still mainly file I/O and
> > processing of the file's contents, just like with byte-compiled files.
> 
> Because a portable dumper can do less, on both file I/O and processing 
> of the file's contents. There's no lisp evaluation, no slurping a whole 
> file into memory. Having to read all of Emacs into memory on startup is 
> a burden even on a fast, modern machine like mine.
> 
> ~/edev/trunk/src
> $ sync && sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
> 
> ~/edev/trunk/src
> $ time pv < emacs >/dev/null
> 48.6MiB 0:00:00 [ 455MiB/s] 
> [=========================================================>] 100% 
> 
> 
> real	0m0.116s

Which is definitely comparable with my measurements of loading all of
the *.elc files concatenated, which were proclaimed to be "too slow".

> > If we have no reason to believe this portable dumper will be
> > significantly faster, we should IMO investigate the .elc method first,
> > because it's so much simpler, both in its implementation and in future
> > maintenance.  E.g., adding a new kind of Lisp object to Emacs would
> > require corresponding changes in the dumper.
> 
> Adding a new kind of lisp object requires changes throughout core 
> anyway.

The changes in the dumper are _in_addition_ to that.

> > Demand paging in an application, and an application such as Emacs on
> > top of that, makes little sense to me.
> 
> Why? It's conceptually no different from autoload.

The devil is in the details, though.  And there are a lot of details
in this case that are completely unrelated to the concept.  If you
don't get them all right, you get a subtly unstable application that
will crash randomly in hard to reproduce and debug situations.

> > This is the OS business, not
> > ours.  Using mmap as a fast way to read a file, yes, that's done in
> > many applications.  But please lets leave demand paging out of our
> > scope.
> 
> Emacs isn't just an application. It's a Lisp virtual machine

No, it's not.  It's an application with a powerful extension language.

> (FWIW, mmap isn't a particularly fast way of doing bulk file reads. 
> That's why GNU grep removed its mmap support.)

It was an example of a low-level technique that is sufficiently simple
to use, that's all.

> > IMO the less we mess with low-level techniques that no other
> > applications use the better, both because we have very few people who
> > can do that and because doing so runs higher risk of becoming broken
> > by future developments in the platforms we deem important.  The
> > long-term tendency in Emacs development should be to move away from
> > such techniques, not to acquire more of them.
> 
> I'm for anything that delivers meaningful performance advantages.

IME, that way lies madness.  It's the exact opposite of the direction
Emacs should evolve if we want to prevent it from becoming a marginal
package for a few enthusiasts.

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

>> At startup, we'll load the dump file and walk the relocations, fixing up
>> all the embedded addresses to account for the new process's different
>> address space.
> Why do you think this will have better performance that reading a
> single .elc file at startup?  It's still mainly file I/O and
> processing of the file's contents, just like with byte-compiled files.

I guess it depends if we can get lread.c to be bound by file-I/O.
Currently, it's significantly slower.

It's clear on the surface that lread.c has more work to do than an ideal
"portable undumper":
- the PU just has to find all pointers and increment them by
  a fixed offset (it could do so either by a GC-like traversal, or by
  consulting an auxiliary precomputed table of addresses stored alongside
  the dump state)
- lread.c has to check every byte for lexing/parsing, it has to call the
  memory allocator for every object, tie the knots for cyclic objects,
  `intern` the symbols, decode the \ in strings, ...

The jury is still out whether this extra work can be implemented
efficiently enough.  There are other differences which can impact the
performance (e.g. the size of the "dump" is likely different in the two
cases, so the amount of I/O is affected) and the desirability (speeding
up loading of .elc would benefit other cases as well, we could generate
a generic dump.elc rather than have it be OS-specific, ...)


        Stefan

Re: Skipping unexec via a big .elc file

[Perry E. Metzger]

On Tue, 25 Oct 2016 18:59:36 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> > Everyone who's seriously thought about the unexec problem
> > _understands_ the issue.
> 
> The important point is that the number of people here who can claim
> such understanding, enough so to fix the issues, is diminishingly
> small, and gets smaller every year.

Just an aside: when you attract fewer and fewer users, you end up with
fewer and fewer contributors. Fewer and fewer contributors makes
maintenance harder and can create a death spiral for projects. If, in
an effort to make maintenance easier, you scare off a lot of users,
you could end up making the maintenance situation worse in the long
run. I'm not saying that longer start time would scare off users as
such, but in general, this balance has to be weighed in making
decisions about usability vs. maintenance costs.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> Date: Tue, 25 Oct 2016 18:53:13 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: Daniel Colascione <dancol@dancol.org>, p.stephani2@gmail.com,
>  monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> On Tue, 25 Oct 2016 18:59:36 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> > > Everyone who's seriously thought about the unexec problem
> > > _understands_ the issue.
> > 
> > The important point is that the number of people here who can claim
> > such understanding, enough so to fix the issues, is diminishingly
> > small, and gets smaller every year.
> 
> Just an aside: when you attract fewer and fewer users, you end up with
> fewer and fewer contributors. Fewer and fewer contributors makes
> maintenance harder and can create a death spiral for projects. If, in
> an effort to make maintenance easier, you scare off a lot of users,
> you could end up making the maintenance situation worse in the long
> run. I'm not saying that longer start time would scare off users as
> such, but in general, this balance has to be weighed in making
> decisions about usability vs. maintenance costs.

That's a profoundly false premise, and a misrepresentation of
everything I wrote.  No one is arguing for slower startup that will
annoy users!  The issue at hand is which approach to prefer when the
startup time is comparable.

Re: Skipping unexec via a big .elc file

[Perry E. Metzger]

On Wed, 26 Oct 2016 05:36:19 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> That's a profoundly false premise, and a misrepresentation of
> everything I wrote.  No one is arguing for slower startup that will
> annoy users!  The issue at hand is which approach to prefer when the
> startup time is comparable.

Good to hear.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 24 Oct 2016 09:17:14 -0700
> 
> > I very much disagree with this: the unexec maintenance situation is
> > actually so fragile that it could break at any moment, in the sense
> > that we could very easily get into having no people on board who know
> > enough about unexec to solve the next problem that will break it.  The
> > number of people who do know gets smaller and smaller with each year.
> > That is not healthy at all for the future of the project.
> 
> In both this discussion and the one about insdel, you've expressed the 
> sentiment that we need to optimize for a world in which very few people 
> have time to maintain Emacs internals. I have a more optimistic view: 
> people are generally good at figuring things out, and if learning about 
> unexec or other esoteric facilities is that prevents a developer from 
> porting Emacs to a new platform or fixing an important bug, that 
> developer will put time into learning about these mechanisms.

Even if you are right, such "figuring out" will take time, and will
delay Emacs development if not stall it.  With enough bad luck, we
could start people abandoning ship.  Like I said, Emacs already cannot
be built on a system with ASLR; how soon do you think this and similar
problems will be considered fatal flaws?

Yes, I'm a pessimist about these aspects of Emacs development.  My
reasons are what I see before my eyes almost every day: some problems
in Emacs are not touched until one of the few who know enough do it.
Look at the last installment of this saga, with ralloc-induced
problems: the same usual suspects are involved in solving it.  If all
of those few were run over by a bus, how fast these problems would be
identified and solved?  And this problem is by far simpler than the
unexec subtleties.  It's no accident that no one (perhaps except Paul)
is seriously working on the unexec replacement.  Why would you believe
that this could change in the future, when most our contributors lack
proficiency working on this level?

So yes, I think your optimism is misplaced.  But that doesn't matter,
because no solution for unexec that is not good enough,
performance-wise and otherwise, will be accepted by the crowd, no
matter how grave is the current situation.  So you should not be
worried about this.  What _is_ important, IMO, is that if and when we
do need to drop unexec, we will have _some_ solution, however
imperfect, to start with and get it up to speed.  Because whatever the
solution, making it happen is a lot of work, and we had better done
most of it by then.

Re: Skipping unexec via a big .elc file

[Perry E. Metzger]

On Mon, 24 Oct 2016 16:35:08 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> > I think some users will run away screaming if Emacs takes a whole
> > second to start up.  
> 
> It depends.  If those users, like me, have hundreds of buffers in
> their sessions, and use desktop.el to recreate their sessions, they
> already wait a few seconds for that.

Just FYI, I find the fact that emacs starts up instantaneously a big
win. If it starts taking a substantial period to start up again like
it did 30 years ago, I'm going to be unhappy. I get that this
simplifies maintenance but the startup time is a big lose.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

> On Oct 23, 2016, at 21:07, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>> That sounds strangely long, as I got less than 2 sec with all the
>> preloaded *.elc files concatenated to a single file, and that's before
>> I made pure-copy a no-op.
>> Another report was that "loadup" with pure-copy short-circuited took
>> less than 0.5 sec.  See
> 
> Hmm... indeed, I got to 0.72s with his patch (on a different, slower
> machine (a Thinkpad X201s, i.e. with a i7 CPU L620 @ 2.00GHz)).
> 
> If I re-add international/characters it goes up a bit to
> 0.96s, but still nowhere near the 3s I got on my big .elc file.
> [ I wonder what makes loading my big file so slow.  ]
> 
> This said, there's still a factor 5-10 to get to "immediate", tho.

I think this came up in the thread Eli referred to, but when I’ve looked at startup time in CANNOT_DUMP builds, a couple of things jumped out at me:

* Garbage collection time.  If we’re not trying to dump out as compact as possible an image, squeezing out every byte is less important.  Drop all of the explicit calls in loadup.el.  Consider raising gc-cons-threshold to the point where it doesn’t trigger during loadup; maybe set it back after startup completes, or the first time Emacs is idle more than a couple seconds.

* I/O processing time — not the I/O system calls, but the C library processing.  Change getc to getc_unlocked in charset.c and lread.c.  (And/or change the loading of dumped.elc to read everything into a buffer and execute code from the buffer, if that might be faster.)  Mutex locking time is costly on Mac OS X, but not exactly free in glibc either.

As I recall, I had startup times under a second without any loadup/dump preprocessing with these changes.  (And all the “purecopy” stuff skipped, in a CANNOT_DUMP build.)

Your “dumped.elc” might trigger some of the same issues.  If the eventual idea is to stuff the “dumped” data into a char array to link into the final installed executable, the second issue is less relevant, though.

Did you check whether actually byte compiling the written file made a difference?

Ken

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

> * Garbage collection time.  If we’re not trying to dump out as compact as
> possible an image, squeezing out every byte is less important.  Drop all of
> the explicit calls in loadup.el.  Consider raising gc-cons-threshold to the
> point where it doesn’t trigger during loadup; maybe set it back after
> startup completes, or the first time Emacs is idle more
> than a couple seconds.

The patch to which I was referring (and which I used) does get rid of
most gc calls.

> As I recall, I had startup times under a second without any loadup/dump
> preprocessing with these changes.  (And all the “purecopy” stuff skipped,
> in a CANNOT_DUMP build.)

The patch also skips the purecopy by setting purify-flag to nil.

> Did you check whether actually byte compiling the written file made
> a difference?

dumped.elc has no code to compile.


        Stefan

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

On Oct 24, 2016, at 09:13, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>> Did you check whether actually byte compiling the written file made
>> a difference?
> 
> dumped.elc has no code to compile.

It has a lot of fset and setplist calls which can be compiled, especially if you reorder things such that they’re not mixed up with the defvar calls that don’t compile.  The generated .elc output is about 25% larger.  I don’t expect the C parts of fset and setplist to be affected at all, of course; the parsing and interpretation of the Lisp may be another matter.  Unfortunately, byte-compile-file doesn’t preserve the sharing of objects (“#42#”) present in the input file, so the output isn’t semantically the same.

I did some profiling.  Without byte compiling, it appears that around half of the CPU time used loading the file in my test is spent in Frassq(…,read_objects), called from substitute_object_recurse.  For processing a file with this much sharing of objects, an assoc list with O(n) access time may not be the best choice.  Whatever we replace it with, it appears we need to be able to look up cons cells in a collection by either element.

The next top users of CPU time (_IO_getc, oblookup) are less significant, though there are some easy minor gains to be made there.

With a hacked-up 31-slot hash table replacing read_objects, the getc_unlocked changes, and setting OBARRAY_SIZE to 8191, I got the load time for the file in batch mode on my test system from just under a half second to about a quarter second.  Nearly half the remaining CPU time is split between readchar, read1, readbyte_from_file, and Fassq.

Ken

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

>>> Did you check whether actually byte compiling the written file made
>>> a difference?
>> dumped.elc has no code to compile.
> It has a lot of fset and setplist calls which can be compiled, especially if
> you reorder things such that they’re not mixed up with the defvar calls that
> don’t compile.

"A lot of" is relative: the time to read them compared to an equivalent
byte-code version should be negligeable, and their execution time should
be even more negligeable.

> The generated .elc output is about 25% larger.

That's not because of byte-compilation per-se.  It's because the
byte-compiler uses `print-circle' but only within each top-level entity,
so you lose sharing between functions and between variables.

IOW you can get the exact same 25% larger file by printing each
fset/defvar/setplist separately (instead of printing them as one big
`progn`).  And you can trick the byte-compiler to preserve this sharing
by replacing the leading `progn` (which the byte-compiler removes) into
a (let () ...), tho maybe you'll need to really add some dummy binding
in that `let` to make sure the byte-compiler doesn't end up removing it.

> I did some profiling.  Without byte compiling, it appears that around half
> of the CPU time used loading the file in my test is spent in
> Frassq(…,read_objects), called from substitute_object_recurse.

Ah, that's what it is.  Clearly we should be able to optimize most of
this away.

> For processing a file with this much sharing of objects, an assoc list with
> O(n) access time may not be the best choice.

Indeed.

> Whatever we replace it with, it appears we need to be able to look up
> cons cells in a collection by either element.

Ideally, we could get rid of substitute_object_in_subtree entirely.
E.g. the patch below skips it for the case of "#n=(...)", and by peeping
ahead to decide the type of placeholder we build, we should be able to
get rid of it in all cases.


        Stefan


diff --git a/src/lread.c b/src/lread.c
index 58d518c..a06a78f 100644
--- a/src/lread.c
+++ b/src/lread.c
@@ -2936,12 +2936,21 @@ read1 (Lisp_Object readcharfun, int *pch, bool first_in_list)
 		      tem = read0 (readcharfun);
 
 		      /* Now put it everywhere the placeholder was...  */
-		      substitute_object_in_subtree (tem, placeholder);
+                      if (CONSP (tem))
+                        {
+                          Fsetcar (placeholder, XCAR (tem));
+                          Fsetcdr (placeholder, XCDR (tem));
+                          return placeholder;
+                        }
+                      else
+                        {
+		          substitute_object_in_subtree (tem, placeholder);
 
-		      /* ...and #n# will use the real value from now on.  */
-		      Fsetcdr (cell, tem);
+		          /* ...and #n# will use the real value from now on.  */
+		          Fsetcdr (cell, tem);
 
-		      return tem;
+		          return tem;
+                        }
 		    }
 
 		  /* #n# returns a previously read object.  */

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

> On Oct 25, 2016, at 09:48, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>>>> Did you check whether actually byte compiling the written file made
>>>> a difference?
>>> dumped.elc has no code to compile.
>> It has a lot of fset and setplist calls which can be compiled, especially if
>> you reorder things such that they’re not mixed up with the defvar calls that
>> don’t compile.
> 
> "A lot of" is relative: the time to read them compared to an equivalent
> byte-code version should be negligeable, and their execution time should
> be even more negligeable.
> 
>> The generated .elc output is about 25% larger.
> 
> That's not because of byte-compilation per-se.  It's because the
> byte-compiler uses `print-circle' but only within each top-level entity,
> so you lose sharing between functions and between variables.
> 
> IOW you can get the exact same 25% larger file by printing each
> fset/defvar/setplist separately (instead of printing them as one big
> `progn`).  And you can trick the byte-compiler to preserve this sharing
> by replacing the leading `progn` (which the byte-compiler removes) into
> a (let () ...), tho maybe you'll need to really add some dummy binding
> in that `let` to make sure the byte-compiler doesn't end up removing it.

Ah, yes… “(let () …)” was enough with no bindings.  Now the compiled file, which now contains only one big byte-code invocation, is still larger than the original dumped file, though not as much, and from a couple of spot checks it looks like the data sharing is indeed preserved.  It also takes longer to load.  Oh well.

> Ideally, we could get rid of substitute_object_in_subtree entirely.
> E.g. the patch below skips it for the case of "#n=(...)", and by peeping
> ahead to decide the type of placeholder we build, we should be able to
> get rid of it in all cases.

I would think not for types using flexible array members, since we may not know the allocation size until we’ve seen the end of the object.


In poking around with gdb, most of the invocations of substitute_object_in_subtree I looked at got a subtree of nil.  It appears to me that if the “subtree” passed isn’t the placeholder and isn’t one of the types we process recursively, then we will never do any substitution, right?  So the checking of seen_list and read_objects isn’t relevant.

I started my tests over with an updated source tree from upstream and put in your loadup.el change.  Running “time emacs -batch -l dumped.elc” took 3.5s; according to “perf record”/“perf report”, Frassq took about 85% of the CPU time, and Fassq took about 9%.

Added your lread.c patch; run time is about 1.8s, 70% in Frassq and almost 20% in Fassq.

Patched substitute_object_recurse after the check for the subtree matching the placeholder, so that if the subtree passed was a symbol or number, it would simply be returned without consulting seen_list or read_objects.  Run time is now 0.7s; Fassq is a bit over 50% of that, and Frassq about 17%, and _IO_getc around 11%.  I think it should be safe to short-circuit it for some other types as well.

I had my getc_unlocked change sitting around so I pulled that in.  Run time is now 0.6s, with Fassq at 57% and Frassq at 18%.

Next on the profiling chart is oblookup, but it’s only at 4% so I’m going to ignore OBARRAY_SIZE for now.  However, OBARRAY_SIZE could affect the order of atoms in processing, which could drastically rearrange the ordering of the data structures in dumped.elc.

I think the next step is to look at replacing read_objects, probably with a pair of hash tables, but it’s getting a bit late for trying that tonight.

Ken

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

I wrote:
> Patched substitute_object_recurse after the check for the subtree matching the placeholder, so that if the subtree passed was a symbol or number, it would simply be returned without consulting seen_list or read_objects.  Run time is now 0.7s; Fassq is a bit over 50% of that, and Frassq about 17%, and _IO_getc around 11%.  I think it should be safe to short-circuit it for some other types as well.
> 
> I had my getc_unlocked change sitting around so I pulled that in.  Run time is now 0.6s, with Fassq at 57% and Frassq at 18%.
> 
> Next on the profiling chart is oblookup, but it’s only at 4% so I’m going to ignore OBARRAY_SIZE for now.  However, OBARRAY_SIZE could affect the order of atoms in processing, which could drastically rearrange the ordering of the data structures in dumped.elc.
> 
> I think the next step is to look at replacing read_objects, probably with a pair of hash tables, but it’s getting a bit late for trying that tonight.

I switched over to a pair of hash tables and the run time is just under 0.2s on my test machine now.  Profiling reports are now topped by read1, readchar, and readbyte_from_file (now including the expanded getc_unlocked calls), accounting for about 30% of the CPU time between them.  The hash functions and substitute_object_recurse are not taking a significant amount of time.

I took a look at the types of shared data in one of the generated dumped.elc files I got; almost 2700 were strings (all without text properties), almost 1900 were cons cells, and the rest numbered under 300.  So I’m not sure special-casing other types besides Lisp_Cons in read1 will gain us much.

It took me a while to sort through the lookups being done during and after parsing of an object and how the checks for circular objects work, but I think I’ve got it working.  I’ve pushed a scratch branch over with the changes if you’d like to try them, though I think I botched the git push syntax when trying to create “scratch/raeburn/startup” somehow, so I created “scratch/raeburn-startup”… or possibly I’ve created both?  I saw an email notification go out for both, but I only see the latter in the repository browser interface… 

Ken

Re: Skipping unexec via a big .elc file

[Simon Leinen]

On Sun, Oct 30, 2016 at 3:43 PM, Ken Raeburn <raeburn@raeburn.org> wrote:
> I switched over to a pair of hash tables and the run time is just under 0.2s on my test machine now.  Profiling reports are now topped by read1, readchar, and readbyte_from_file (now including the expanded getc_unlocked calls), accounting for about 30% of the CPU time between them.  The hash functions and substitute_object_recurse are not taking a significant amount of time. [...]

Promising!

Years ago I spent some time optimizing the MIB-reading code in
UCD/Net-SNMP, and found that the biggest win was to treat the input
file as one big buffer (I actually mmap()ped it) and then avoid most
of the memory allocation overhead of token creation by using start/end
pointers directly into that buffer.  I never upstreamed that code, and
I'm not sure the representation would have been acceptable to the
other developers.  But it sure was fast.  Maybe an approach like that
would be suitable for .elc loading.
-- 
Simon.

Re: Skipping unexec via a big .elc file

[Daniel Colascione]

On 10/30/2016 07:43 AM, Ken Raeburn wrote:
>
> I wrote:
>> Patched substitute_object_recurse after the check for the subtree matching the placeholder, so that if the subtree passed was a symbol or number, it would simply be returned without consulting seen_list or read_objects.  Run time is now 0.7s; Fassq is a bit over 50% of that, and Frassq about 17%, and _IO_getc around 11%.  I think it should be safe to short-circuit it for some other types as well.
>>
>> I had my getc_unlocked change sitting around so I pulled that in.  Run time is now 0.6s, with Fassq at 57% and Frassq at 18%.
>>
>> Next on the profiling chart is oblookup, but it’s only at 4% so I’m going to ignore OBARRAY_SIZE for now.  However, OBARRAY_SIZE could affect the order of atoms in processing, which could drastically rearrange the ordering of the data structures in dumped.elc.
>>
>> I think the next step is to look at replacing read_objects, probably with a pair of hash tables, but it’s getting a bit late for trying that tonight.
>
> I switched over to a pair of hash tables and the run time is just under 0.2s on my test machine now.  Profiling reports are now topped by read1, readchar, and readbyte_from_file (now including the expanded getc_unlocked calls), accounting for about 30% of the CPU time between them.  The hash functions and substitute_object_recurse are not taking a significant amount of time.
>
> I took a look at the types of shared data in one of the generated dumped.elc files I got; almost 2700 were strings (all without text properties), almost 1900 were cons cells, and the rest numbered under 300.  So I’m not sure special-casing other types besides Lisp_Cons in read1 will gain us much.
>
> It took me a while to sort through the lookups being done during and after parsing of an object and how the checks for circular objects work, but I think I’ve got it working.  I’ve pushed a scratch branch over with the changes if you’d like to try them, though I think I botched the git push syntax when trying to create “scratch/raeburn/startup” somehow, so I created “scratch/raeburn-startup”… or possibly I’ve created both?  I saw an email notification go out for both, but I only see the latter in the repository browser interface…
>
> Ken

Awesome! Even if we go for something besides a big elc file for startup, 
these improvements will help.

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

> I switched over to a pair of hash tables and the run time is just under 0.2s
> on my test machine now.  Profiling reports are now topped by read1,
> readchar, and readbyte_from_file (now including the expanded getc_unlocked
> calls), accounting for about 30% of the CPU time between them.  The hash
> functions and substitute_object_recurse are not taking a significant amount
> of time.

BTW, I don't know if you've tried to make that dumped file work
correctly, but in case you haven't here's my latest attempt.

It mostly works, tho there are still issues such as the fact that the
global-font-lock-mode still fails to be properly enabled.


        Stefan


diff --git a/lisp/emacs-lisp/macroexp.el b/lisp/emacs-lisp/macroexp.el
index 310ca29..9ca53eb 100644
--- a/lisp/emacs-lisp/macroexp.el
+++ b/lisp/emacs-lisp/macroexp.el
@@ -439,7 +439,8 @@ macroexp--const-symbol-p
   (or (memq symbol '(nil t))
       (keywordp symbol)
       (if any-value
-	  (or (memq symbol byte-compile-const-variables)
+	  (or (and (boundp 'byte-compile-const-variables)
+                   (memq symbol byte-compile-const-variables))
 	      ;; FIXME: We should provide a less intrusive way to find out
 	      ;; if a variable is "constant".
 	      (and (boundp symbol)
diff --git a/lisp/international/mule.el b/lisp/international/mule.el
index 21ab7e1..bb4808b 100644
--- a/lisp/international/mule.el
+++ b/lisp/international/mule.el
@@ -290,7 +290,7 @@ define-charset
 		      elt))
 		  props))
     (setcdr (assq :plist attrs) props)
-
+    (put name 'internal--charset-args (mapcar #'cdr attrs))
     (apply 'define-charset-internal name (mapcar 'cdr attrs))))
 
 
@@ -911,6 +911,8 @@ define-coding-system
 	  (cons :name (cons name (cons :docstring (cons (purecopy docstring)
 							props)))))
     (setcdr (assq :plist common-attrs) props)
+    (put name 'internal--cs-args
+         (mapcar #'cdr (append common-attrs spec-attrs)))
     (apply 'define-coding-system-internal
 	   name (mapcar 'cdr (append common-attrs spec-attrs)))))
 
diff --git a/lisp/loadup.el b/lisp/loadup.el
index 21c64a8..5967334 100644
--- a/lisp/loadup.el
+++ b/lisp/loadup.el
@@ -1,4 +1,4 @@
-;;; loadup.el --- load up standardly loaded Lisp files for Emacs
+;;; loadup.el --- load up standardly loaded Lisp files for Emacs  -*- lexical-binding:t -*-
 
 ;; Copyright (C) 1985-1986, 1992, 1994, 2001-2016 Free Software
 ;; Foundation, Inc.
@@ -461,6 +461,150 @@
 						invocation-directory)
 			      (expand-file-name name invocation-directory)
 			      t)))
+      (message "Dumping into dumped.elc...preparing...")
+
+      ;; Dump the current state into a file so we can reload it!
+      (message "Dumping into dumped.elc...generating...")
+      (let ((faces '())
+            (coding-systems '()) (coding-system-aliases '())
+            (charsets '()) (charset-aliases '())
+            (cmds '()))
+        (setcdr global-buffers-menu-map nil) ;; Get rid of buffer objects!
+        (mapatoms
+         (lambda (s)
+           (when (fboundp s)
+             (if (subrp (symbol-function s))
+                 ;; subr objects aren't readable!
+                 (unless (equal (symbol-name s) (subr-name (symbol-function s)))
+                   (push `(fset ',s (symbol-function ',(intern (subr-name (symbol-function s))))) cmds))
+               (if (memq s '(rename-buffer))
+                   ;; FIXME: We need these, but they contain
+                   ;; unprintable objects.
+                   nil
+                 (push `(fset ',s ,(macroexp-quote (symbol-function s)))
+                       cmds))))
+           (when (and (boundp s)
+                      (not (macroexp--const-symbol-p s 'any-value))
+                      ;; I think we don't need/want these!
+                      (not (memq s '(terminal-frame obarray
+                                     initial-window-system window-system
+                                     ;; custom-delayed-init-variables
+                                     exec-path
+                                     process-environment
+                                     command-line-args noninteractive))))
+             ;; FIXME: Handle varaliases!
+             (let ((v (symbol-value s)))
+               (push `(set-default
+                       ',s
+                       ,(cond
+                         ;; FIXME: (Correct) hack to avoid
+                         ;; unprintable objects.
+                         ((eq s 'undo-auto--undoably-changed-buffers) nil)
+                         ;; FIXME: Incorrect hack to avoid
+                         ;; unprintable objects.
+                         ((eq s 'advertised-signature-table)
+                          (make-hash-table :test 'eq :weakness 'key))
+                         ((subrp v)
+                          `(symbol-function ',(intern (subr-name v))))
+                         ((and (markerp v) (null (marker-buffer v)))
+                          '(make-marker))
+                         ((and (overlayp v) (null (overlay-buffer v)))
+                          '(let ((ol (make-overlay (point-min) (point-min))))
+                             (delete-overlay ol)
+                             ol))
+                         (v (macroexp-quote v))))
+                     cmds)
+               (push `(defvar ,s) cmds)))
+           (when (symbol-plist s)
+             (push `(setplist ',s ',(symbol-plist s)) cmds))
+           (when (get s 'face-defface-spec)
+             (push s faces))
+           (if (get s 'internal--cs-args)
+               (push s coding-systems))
+           (when (and (coding-system-p s)
+                      (not (eq s (car (coding-system-aliases s)))))
+             (push (cons s (car (coding-system-aliases s)))
+                   coding-system-aliases))
+           (if (get s 'internal--charset-args)
+               (push s charsets)
+             (when (and (charsetp s)
+                        (not (eq s (get-charset-property s :name))))
+               (push (cons s (get-charset-property s :name))
+                     charset-aliases))))
+         obarray)
+        (message "Dumping into dumped.elc...printing...")
+        (with-current-buffer (generate-new-buffer "dumped.elc")
+          (insert ";ELC\^W\^@\^@\^@\n;;; Compiled\n;;; in Emacs version "
+                  emacs-version "\n")
+          (let ((print-circle t)
+                (print-gensym t)
+                (print-quoted t)
+                (print-level nil)
+                (print-length nil)
+                (print-escape-newlines t)
+                (standard-output (current-buffer)))
+            (print `(progn . ,cmds))
+            (terpri)
+            (print `(let ((css ',charsets))
+                      (dotimes (i 3)
+                        (dolist (cs (prog1 css (setq css nil)))
+                          ;; (message "Defining charset %S..." cs)
+                          (condition-case nil
+                              (progn
+                                (apply #'define-charset-internal
+                                       cs (get cs 'internal--charset-args))
+                                ;; (message "Defining charset %S...done" cs)
+                                )
+                            (error
+                             ;; (message "Defining charset %S...postponed"
+                             ;;          cs)
+                             (push cs css)))))))
+            (terpri)
+            (print `(dolist (cs ',charset-aliases)
+                      (define-charset-alias (car cs) (cdr cs))))
+            (terpri)
+            (print `(let ((css ',coding-systems))
+                      (dotimes (i 3)
+                        (dolist (cs (prog1 css (setq css nil)))
+                          ;; (message "Defining coding-system %S..." cs)
+                          (condition-case nil
+                              (progn
+                                (apply #'define-coding-system-internal
+                                       cs (get cs 'internal--cs-args))
+                                ;; (message "Defining coding-system %S...done" cs)
+                                )
+                            (error
+                             ;; (message "Defining coding-system %S...postponed"
+                             ;;          cs)
+                             (push cs css)))))))
+            (print `(dolist (f ',faces)
+                      (face-spec-set f (get f 'face-defface-spec)
+                                     'face-defface-spec)))
+            (terpri)
+            (print `(dolist (cs ',coding-system-aliases)
+                      (define-coding-system-alias (car cs) (cdr cs))))
+            (terpri)
+            (print `(progn
+                      ;; (message "Done preloading!")
+                      ;; (message "custom-delayed-init-variables = %S"
+                      ;;          custom-delayed-init-variables)
+                      ;; (message "Running top-level = %S" top-level)
+                      (setq debug-on-error t)
+                      (use-global-map global-map)
+                      (eval top-level)
+                      ;; (message "top-level done!?")
+                      ))
+            (terpri))
+          (goto-char (point-min))
+          (while (re-search-forward " (\\(defvar\\|setplist\\|fset\\) " nil t)
+            (goto-char (match-beginning 0))
+            (delete-char 1) (insert "\n"))
+          (message "Dumping into dumped.elc...saving...")
+          (let ((coding-system-for-write 'emacs-internal))
+            (write-region (point-min) (point-max) (buffer-name)))
+          (message "Dumping into dumped.elc...done")
+          ))
+
       (kill-emacs)))
 
 ;; For machines with CANNOT_DUMP defined in config.h,
diff --git a/src/coding.c b/src/coding.c
index 9f709be..a677758 100644
--- a/src/coding.c
+++ b/src/coding.c
@@ -10326,8 +10326,9 @@ usage: (define-coding-system-internal ...)  */)
       CHECK_NUMBER_CAR (reg_usage);
       CHECK_NUMBER_CDR (reg_usage);
 
-      request = Fcopy_sequence (args[coding_arg_iso2022_request]);
-      for (tail = request; CONSP (tail); tail = XCDR (tail))
+      request = Qnil;
+      for (tail = args[coding_arg_iso2022_request];
+            CONSP (tail); tail = XCDR (tail))
 	{
 	  int id;
 	  Lisp_Object tmp1;
@@ -10339,7 +10340,8 @@ usage: (define-coding-system-internal ...)  */)
 	  CHECK_NATNUM_CDR (val);
 	  if (XINT (XCDR (val)) >= 4)
 	    error ("Invalid graphic register number: %"pI"d", XINT (XCDR (val)));
-	  XSETCAR (val, make_number (id));
+	  request = Fcons (Fcons (make_number (id), XCDR (val)),
+                           request);
 	}
 
       flags = args[coding_arg_iso2022_flags];
diff --git a/src/emacs.c b/src/emacs.c
index 2480dfc..bdf3742 100644
--- a/src/emacs.c
+++ b/src/emacs.c
@@ -1593,9 +1593,9 @@ Using an Emacs configured with --with-x-toolkit=lucid does not have this problem
 #endif
 	  Vtop_level = list2 (Qload, build_unibyte_string (file));
 	}
-      /* Unless next switch is -nl, load "loadup.el" first thing.  */
-      if (! no_loadup)
-	Vtop_level = list2 (Qload, build_string ("loadup.el"));
+      else if (! no_loadup)
+        /* Unless next switch is -nl, load "loadup.el" first thing.  */
+	Vtop_level = list2 (Qload, build_string ("../src/dumped.elc"));
     }
 
   /* Set up for profiling.  This is known to work on FreeBSD,

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

On Oct 31, 2016, at 10:27, Stefan Monnier <monnier@iro.umontreal.ca> wrote:


>> I switched over to a pair of hash tables and the run time is just under 0.2s
>> on my test machine now.  Profiling reports are now topped by read1,
>> readchar, and readbyte_from_file (now including the expanded getc_unlocked
>> calls), accounting for about 30% of the CPU time between them.  The hash
>> functions and substitute_object_recurse are not taking a significant amount
>> of time.
> 
> BTW, I don't know if you've tried to make that dumped file work
> correctly, but in case you haven't here's my latest attempt.

Thanks!  Looks like you’ve refined the handling of faces and other attributes.  Have you tried it out in batch mode?  I’m getting a crash in realize_face with a null cache pointer.

Ken

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

> Thanks!  Looks like you’ve refined the handling of faces and other
> attributes.  Have you tried it out in batch mode?

No, haven't gotten that far.


        Stefan

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

BTW, it might be worth comparing the behavior with the one we get with
the "normal" temacs (i.e. by loading loadup.el instead of dumped.elc),
as well as with what we get with CANNOT_DUMP: from what I remember the
current code doesn't handle CANNOT_DUMP 100% correctly (which is OK so
far because CANNOT_DUMP is only ever used temporarily during porting
until unexec is working).

IOW some of the problems we may encounter could be unrelated to what we
do w.r.t dumped.elc.


        Stefan

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

On Nov 2, 2016, at 08:22, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> BTW, it might be worth comparing the behavior with the one we get with
> the "normal" temacs (i.e. by loading loadup.el instead of dumped.elc),
> as well as with what we get with CANNOT_DUMP: from what I remember the
> current code doesn't handle CANNOT_DUMP 100% correctly (which is OK so
> far because CANNOT_DUMP is only ever used temporarily during porting
> until unexec is working).

…which is why it seems like I have to keep fixing bugs every time I try to use it.

If CANNOT_DUMP mode worked reliably, I’d think it would be the logical starting point for this work — it’s already compiling with the expectation that we’ll be loading the Lisp code when the user starts it up rather than preparing for unexec and a second invocation of main().  Changing the thing we load at startup from loadup.el to dumped.elc should be pretty minor.

I’m trying CANNOT_DUMP out right now.  As soon as I’ve got it bootstrapping again, I’ll try pulling in the other changes.

> IOW some of the problems we may encounter could be unrelated to what we
> do w.r.t dumped.elc.

True.

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

I’ve pushed an update to the scratch/raeburn-startup branch.  It includes several updates:

* Stefan’s Oct 31 patch instead of his earlier one.  This does more reinitializing of charsets, coding systems, etc., which I believe were absent from the previous version.

* More patches to the recursive object substitution pass done during reading.  The big costs on Mac OS X seem to differ from my Linux/GNU/X11 build — there’s a much larger dumped.elc file, and an entirely different compiler — but I’ve managed to trim the run time there a bit.

* Changed gc-cons-threshold to be much larger.  By itself, this isn’t a good change.  But we’d exceed the old value many times over just reading the big “progn” form; this way my Linux/GNU/X11 run doesn’t trigger GC during startup, though I think the Mac version still does.  I think a better strategy might try to defer or discourage GC during startup, and do it instead when we have idle cycles while the user isn’t trying to get something done.  But revamping the GC strategy is a different discussion.

* Larger obarray.  After startup, my Linux/GNU/X11 build has over 15k symbols, and my Mac build has over 21k.  The old obarray size of 1511 meant average chain lengths of over 10 and 14.  Shorter chains mean less time spent in oblookup.  And extra slots are cheap.

* Open-code reading ASCII symbol characters from a file in read1().  The hot path involved examining readcharfun to determine its type, compare it against some known symbols, select a function to call, have that function check to see if we’re doing pushback instead of actually reading, block input, do the actual getc() call, and unblock input — all for each character.  The new version duplicates a bunch of code, but once it sees we’re reading from a file, skips most of that for the common path through the inner loop.  This cut maybe 10% off of some of my run times.

With all these changes — Stefan’s new patch with additional initialization, and my updates to shave a little more time off — I’m still hitting just under 0.2s for:

  time ./temacs --batch --eval '(progn (message "hi") (kill-emacs))'

on Linux/GNU/X11 (Intel Core i5-2320, 3GHz, gcc 4.9); my Mac (Intel Core 2 Duo, 2.8GHz) takes over half a second (including at least one GC invocation).

It can be tested by running “temacs” after building it.  The lisp load path will be set based on the source tree, not the installation prefix.  If “-nl” and “-l” arguments are not given, it’ll load “../src/dumped.elc”, but that’s interpreted relative to the lisp *source* directory.  If you build in a directory other than the checked-out tree (i.e., $srcdir is not “.”) as I do, you’ll need to copy dumped.elc from the src directory of the build tree where it’s generated to the src directory of the source tree where it’s sought.

If dumped.elc isn’t found, temacs will exit with status 42.  Under Stefan’s version, an X11 run would spit out a message saying the file wasn’t found and exit, but a tty run would get into a loop complaining about internal-echo-keystrokes-prefix and would need to be killed from another terminal.  This way, it only kind of sucks, equally in both cases. :-)

The remaining time still seems to be about 2/3 reading and parsing bytes, allocating objects, and updating (mostly scanning) the obarray.  There should be a bit more time that can be squeezed out.

Ken

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Sun, 11 Dec 2016 08:34:01 -0500
> 
> I’ve pushed an update to the scratch/raeburn-startup branch.  It includes several updates:

Thank you for your work, I will definitely try to check it out soon.

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> Date: Sun, 11 Dec 2016 17:42:21 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > From: Ken Raeburn <raeburn@raeburn.org>
> > Date: Sun, 11 Dec 2016 08:34:01 -0500
> > 
> > I’ve pushed an update to the scratch/raeburn-startup branch.  It includes several updates:
> 
> Thank you for your work, I will definitely try to check it out soon.

I took a quick look.  There are a few issues with the Windows build
there, but I have one question to which I'd like to know the answer
first: why do we still dumping temacs to emacs, instead of loading
dumped.elc into a bare emacs?

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

> I took a quick look.  There are a few issues with the Windows build
> there, but I have one question to which I'd like to know the answer
> first: why do we still dumping temacs to emacs, instead of loading
> dumped.elc into a bare emacs?

I don't know if Ken has another reason, but in my case it's simply
because I haven't bothered to change the code so as not to call the
unexec dump.


        Stefan

Re: Skipping unexec via a big .elc file

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > * Changed gc-cons-threshold to be much larger.

How about binding it to a higher value for loadup?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

> On Dec 11, 2016, at 14:18, Richard Stallman <rms@gnu.org> wrote:
> 
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>> * Changed gc-cons-threshold to be much larger.
> 
> How about binding it to a higher value for loadup?

That may be good enough.  But GC will probably kick in right after we set it back, so probably most methods we might try for measuring startup time will incur the cost of at least one GC pass, and it’ll happen when the user starts Emacs in real life.  I guess one question is, how much it matters?  It’s only a fraction of a second, but I’m trying to shave a startup time of 0.2s (or 0.6s on Mac OS X) down closer to 0.1s, so a fraction of a second can make a difference.

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Thu, 15 Dec 2016 07:57:09 -0500
> Cc: emacs-devel@gnu.org
> 
> > How about binding it to a higher value for loadup?
> 
> That may be good enough.  But GC will probably kick in right after we set it back

AFAIK, just setting the GC threshold doesn't automatically invoke GC,
you need do something that calls maybe_gc.

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

> On Dec 15, 2016, at 11:04, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Ken Raeburn <raeburn@raeburn.org>
>> Date: Thu, 15 Dec 2016 07:57:09 -0500
>> Cc: emacs-devel@gnu.org
>> 
>>> How about binding it to a higher value for loadup?
>> 
>> That may be good enough.  But GC will probably kick in right after we set it back
> 
> AFAIK, just setting the GC threshold doesn't automatically invoke GC,
> you need do something that calls maybe_gc.

Right, but if we’re not following it up with evaluating another form from dumped.elc (eval_sub can invoke GC) or invoking some compiled routine (branch operations can invoke GC), then we’re probably ready to check for availability of user input (which can invoke GC).

Re: Skipping unexec via a big .elc file

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > * Larger obarray.  After startup, my Linux/GNU/X11 build has over
  > * 15k symbols, and my Mac build has over 21k.  The old obarray
  > * size of 1511 meant average chain lengths of over 10 and 14.
  > * Shorter chains mean less time spent in oblookup.  And extra
  > * slots are cheap.

This may be a good idea, but it has nothing to do with any particular
method of startup or dumping.  So how about doing it unconditionally?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

> On Dec 11, 2016, at 14:18, Richard Stallman <rms@gnu.org> wrote:
> 
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>> * Larger obarray.  After startup, my Linux/GNU/X11 build has over
>> * 15k symbols, and my Mac build has over 21k.  The old obarray
>> * size of 1511 meant average chain lengths of over 10 and 14.
>> * Shorter chains mean less time spent in oblookup.  And extra
>> * slots are cheap.
> 
> This may be a good idea, but it has nothing to do with any particular
> method of startup or dumping.  So how about doing it unconditionally?

A few of the changes on this branch would probably improve speed at least a tiny bit regardless of the startup method.  This one also has the advantage of being a trivial change with, as far as I can see, no down side, so, yeah….

Ken

Re: Skipping unexec via a big .elc file

[Ken Brown]

On 12/11/2016 8:34 AM, Ken Raeburn wrote:
> I’ve pushed an update to the scratch/raeburn-startup branch.

Did you actually push these changes?  The last commit I see at 
http://git.savannah.gnu.org/cgit/emacs.git/log/?h=scratch/raeburn-startup 
is dated 2016-10-30.

Ken

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

> On Dec 13, 2016, at 10:21, Ken Brown <kbrown@cornell.edu> wrote:
> 
> On 12/11/2016 8:34 AM, Ken Raeburn wrote:
>> I’ve pushed an update to the scratch/raeburn-startup branch.
> 
> Did you actually push these changes?  The last commit I see at http://git.savannah.gnu.org/cgit/emacs.git/log/?h=scratch/raeburn-startup is dated 2016-10-30.
> 
> Ken

Strange, I’m not sure what happened.  I’ll push it again.  Just as well, I’ve got a couple minor updates to add anyway.

Ken

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

> On Dec 14, 2016, at 00:30, Ken Raeburn <raeburn@raeburn.org> wrote:
> 
> 
>> On Dec 13, 2016, at 10:21, Ken Brown <kbrown@cornell.edu> wrote:
>> 
>> On 12/11/2016 8:34 AM, Ken Raeburn wrote:
>>> I’ve pushed an update to the scratch/raeburn-startup branch.
>> 
>> Did you actually push these changes?  The last commit I see at http://git.savannah.gnu.org/cgit/emacs.git/log/?h=scratch/raeburn-startup is dated 2016-10-30.
>> 
>> Ken
> 
> Strange, I’m not sure what happened.  I’ll push it again.  Just as well, I’ve got a couple minor updates to add anyway.

I must have overlooked it the first time, but “git push -f” is being rejected with:

remote: error: denying non-fast-forward refs/heads/scratch/raeburn-startup (you should pull first)

Ken

Re: Skipping unexec via a big .elc file

[Phil Sainty]

On 14/12/16 18:45, Ken Raeburn wrote:
> I must have overlooked it the first time, but “git push -f” is being
rejected with:
> remote: error: denying non-fast-forward
refs/heads/scratch/raeburn-startup (you should pull first)

Which typically means that you've amended or rebased your local
history since you last pushed it. Or potentially someone else has
pushed to that branch in the interim. Your local branch and the
remote branch have diverged, at any rate.

If you *really really* want to push a revised history -- bearing
in mind that it will cause merge issues for anyone who has already
pulled from that branch (which is why you would usually refrain
from doing such a thing, and why server-side protection like this
exists in the first place) -- then you can generally work around
the protection by deleting the upstream branch and then pushing
your new version of it.

Alternatively, you may need to rebase your local branch onto the
upstream revision.

Re: Skipping unexec via a big .elc file

[Yuri Khan]

On Wed, Dec 14, 2016 at 5:58 PM, Phil Sainty <psainty@orcon.net.nz> wrote:

> If you *really really* want to push a revised history -- bearing
> in mind that it will cause merge issues for anyone who has already
> pulled from that branch (which is why you would usually refrain
> from doing such a thing, and why server-side protection like this
> exists in the first place) -- then you can generally work around
> the protection by deleting the upstream branch and then pushing
> your new version of it.

Scratch branches are personal and others should exercise caution
if/when pulling them. It is okay for the branch owner to delete and
recreate it.

Re: Skipping unexec via a big .elc file

[Lars Ingebrigtsen]

Ken Raeburn <raeburn@raeburn.org> writes:

> I must have overlooked it the first time, but “git push -f” is being
> rejected with:
>
> remote: error: denying non-fast-forward
> refs/heads/scratch/raeburn-startup (you should pull first)

Sounds like you need to say "git pull --rebase" before you push.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

Branch scratch/raeburn-startup deleted and re-pushed.

In addition to the changes I mentioned earlier, I found an unnecessary memset in the face reinitialization code that could go, and an initialization form was being emitted that tried to incorporate the obarray by value (which wouldn’t work because the symbol chains don’t all get dumped); omitting the latter for now cuts the file size a percent or so.

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

One area I’m contemplating now is whether we can trim the size of dumped.elc.

Question: How useable does Emacs need to be if the Lisp code is improperly installed (or not installed) and can’t be loaded?

With the big-elc approach as currently implemented, assuming we store dumped.elc in the install tree along with the other Lisp code, it basically can’t start without that Lisp library.

If that’s okay, then the next question is: How much do we *need* to load before processing user input?

My impression has been that the current loadup.el contents cover not just the bare minimum that Emacs absolutely needs to have to function, but also the popular things we want to have readily available without having to wait for a Lisp package to load (like buff-menu), especially if it’s in response to a simple keypress or mouse click.  (The X support code is loaded in an X build, even if no X display is present at startup; I think parts of it fall into both categories.)  And adding stuff is fairly cheap; loadup and unexec can take as long as they want, only the speed of relaunching the resulting executable affects the user.

With the big-elc approach, the tradeoffs change.  Reading a bunch of function definitions from dumped.elc is only a tiny bit faster than reading the same definitions from the original .elc files, because we don’t have to open more files.  (At least, the cost is trivial if the files are in cache.  It’ll be OS- and system-dependent.)  Only the time for precomputation that gets done as the file is loaded is saved, and exchanged for the time needed to parse the saved result.

If we can trim some stuff from loadup.el, and resort to autoloading that stuff later, that may save us some startup time.  (Some text mode commands?  Or buff-menu?)

If we really want some of the other stuff to be able to run immediately when the user hits a key, maybe there’s some way to compromise between that and faster startup.  A strawman proposal: load the “must-haves” via dumped.elc; load the user’s init file, read files and execute eval commands as indicated by the command line options; check for user input; if there’s no user input (e.g., use an idle timer set for 3s), start going through a list of “nice-to-haves” and loading them, continuing until user input is available.  If the user starts typing or otherwise invoking some of those nice-to-have commands right away, they’ll have to wait while autoloading happens, but if we get a couple idle seconds, we may still pull the commands in before the user needs them.

Of course, if the user types something while we’re loading a file, that file will have to finish loading before we can respond; it’s sort of a guessing game as to how much idle time suggests that the user is doing something else and probably won’t type anything in the next second or two.  Perhaps we can divide the task further to keep any individual delay shorter: read a .elc file into a buffer, check for input, parse into S-expressions, check for input, eval the S-expressions…

Ken

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Thu, 15 Dec 2016 12:28:15 -0500
> 
> Question: How useable does Emacs need to be if the Lisp code is improperly installed (or not installed) and can’t be loaded?

I had this same idea just the other day.  We have auto-loading, so I
went, so maybe just starting temacs and letting it load whatever it
needs when it needs that would be good enough?

Just to see what would we be up against, I ran

  ./temacs -Q -nl

and sure thing, it errored out right away because some C code called
Lisp which wasn't loaded yet.

What's more, auto-loading doesn't work for preloaded packages, because
we have code in autoload.el to skip/ignore autoload cookies in files
mentioned in loadup.el.

So my next idea would be to come up with a smaller loadup.el which
only loads the stuff that is needed for temacs to start.  I didn't try
that yet, but I did think that Phillip's work on ldefs-boot might just
be a good starting point: those ldefs-boot-*.el files might be just
what we need.

IMO, it would be interesting to see where this will take us, and what
kind of performance could that produce.

Thanks.

Re: Skipping unexec via a big .elc file

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 437 bytes --]

On 2016-12-15 14:59, Eli Zaretskii wrote:
> IMO, it would be interesting to see where this will take us, and what
> kind of performance could that produce.

This sounds like a good idea; I wonder how much it will break, though.  Many external packages don't (require) preloaded packages (some preloaded packages don't or used to not export a (provide), in fact), which may cause issues if these packages aren't preloaded anymore.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Thu, 15 Dec 2016 17:07:50 -0500
> 
> On 2016-12-15 14:59, Eli Zaretskii wrote:
> > IMO, it would be interesting to see where this will take us, and what
> > kind of performance could that produce.
> 
> This sounds like a good idea; I wonder how much it will break, though.  Many external packages don't (require) preloaded packages (some preloaded packages don't or used to not export a (provide), in fact), which may cause issues if these packages aren't preloaded anymore.

Autoloading should fix that.  This idea won't work anyway without
adding the relevant symbols to loaddefs.el.

Re: Skipping unexec via a big .elc file

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 944 bytes --]

On 2016-12-16 02:54, Eli Zaretskii wrote:
>> From: Clément Pit--Claudel <clement.pit@gmail.com>
>> Date: Thu, 15 Dec 2016 17:07:50 -0500
>>
>> On 2016-12-15 14:59, Eli Zaretskii wrote:
>>> IMO, it would be interesting to see where this will take us, and what
>>> kind of performance could that produce.
>>
>> This sounds like a good idea; I wonder how much it will break, though.  Many external packages don't (require) preloaded packages (some preloaded packages don't or used to not export a (provide), in fact), which may cause issues if these packages aren't preloaded anymore.
> 
> Autoloading should fix that.  This idea won't work anyway without
> adding the relevant symbols to loaddefs.el.

Indeed; but then we need to autoload all functions in these files, right?
Also, does autoloading work for macros? And would there not be potential issues with variables and/or defcustoms? (I can't think of any)

Clément.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Fri, 16 Dec 2016 09:28:37 -0500
> 
> > Autoloading should fix that.  This idea won't work anyway without
> > adding the relevant symbols to loaddefs.el.
> 
> Indeed; but then we need to autoload all functions in these files, right?

Not sure about "all", but most of them, yes.  And variables.

> Also, does autoloading work for macros?

It doesn't, but why would macros be a problem?  They need to be seen
by the byte compiler when it compiles the package, not when the
package is loaded.

Re: Skipping unexec via a big .elc file

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 806 bytes --]

On 2016-12-16 09:39, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org
>> From: Clément Pit--Claudel <clement.pit@gmail.com>
>> Date: Fri, 16 Dec 2016 09:28:37 -0500
>>
>>> Autoloading should fix that.  This idea won't work anyway without
>>> adding the relevant symbols to loaddefs.el.
>>
>> Indeed; but then we need to autoload all functions in these files, right?
> 
> Not sure about "all", but most of them, yes.  And variables.
> 
>> Also, does autoloading work for macros?
> 
> It doesn't, but why would macros be a problem?  They need to be seen
> by the byte compiler when it compiles the package, not when the
> package is loaded.

Right; but won't we have a problem when package.el compiles newly downloaded packages that depend on formerly autoloaded libraries?

Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Fri, 16 Dec 2016 10:28:22 -0500
> 
> >> Also, does autoloading work for macros?
> > 
> > It doesn't, but why would macros be a problem?  They need to be seen
> > by the byte compiler when it compiles the package, not when the
> > package is loaded.
> 
> Right; but won't we have a problem when package.el compiles newly downloaded packages that depend on formerly autoloaded libraries?

I don't know.  Maybe.  Determining this would be part of the job of
exploring this alternative.

Re: Skipping unexec via a big .elc file

[Noam Postavsky]

On Fri, Dec 16, 2016 at 4:27 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Cc: emacs-devel@gnu.org
>> From: Clément Pit--Claudel <clement.pit@gmail.com>
>> Date: Fri, 16 Dec 2016 10:28:22 -0500
>>
>> >> Also, does autoloading work for macros?
>> >
>> > It doesn't, but why would macros be a problem?  They need to be seen
>> > by the byte compiler when it compiles the package, not when the
>> > package is loaded.

Why do we think autoloading doesn't work for macros?

autoload is a built-in function in `C source code'.

(autoload FUNCTION FILE &optional DOCSTRING INTERACTIVE TYPE)
[...]
Fifth arg TYPE indicates the type of the object:
[...]
   `macro' or t says FUNCTION is really a macro.


>>
>> Right; but won't we have a problem when package.el compiles newly downloaded packages that depend on formerly autoloaded libraries?
>
> I don't know.  Maybe.  Determining this would be part of the job of
> exploring this alternative.
>

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

>> Also, does autoloading work for macros?
> It doesn't, but why would macros be a problem?

Huh?  Autoloading works fine for macros.  Or maybe I'm misundertanding
Clement's question.

The problem of autoloading is with variables, coding systems, faces, ...
This said, I don't see any strong reason why we couldn't arrange to
autoload coding systems.

Maybe we should instrument the Emacs code to mark functions that are
called during a normal "start Emacs opening a file in fundamental mode".
Then we can look at the preloaded functions&macros which haven't been
used.  This should give us a good idea of how much there is to gain on
this front.


        Stefan

Re: Skipping unexec via a big .elc file

[Phillip Lord]

Clément Pit--Claudel <clement.pit@gmail.com> writes:

> On 2016-12-16 02:54, Eli Zaretskii wrote:
>>> From: Clément Pit--Claudel <clement.pit@gmail.com>
>>> Date: Thu, 15 Dec 2016 17:07:50 -0500
>>>
>>> On 2016-12-15 14:59, Eli Zaretskii wrote:
>>>> IMO, it would be interesting to see where this will take us, and what
>>>> kind of performance could that produce.
>>>
>>> This sounds like a good idea; I wonder how much it will break, though.
>>> Many external packages don't (require) preloaded packages (some preloaded
>>> packages don't or used to not export a (provide), in fact), which may cause
>>> issues if these packages aren't preloaded anymore.
>> 
>> Autoloading should fix that.  This idea won't work anyway without
>> adding the relevant symbols to loaddefs.el.
>
> Indeed; but then we need to autoload all functions in these files, right?


Unfortunately, there is a fairly random aspect to it. You need to
autoload the first function in the file that actually gets called. Once
the file is actually loaded, autoloads don't matter any more. Obviously,
though, which the first function actually is may change.

Phil

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> Date: Thu, 15 Dec 2016 21:59:08 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> So my next idea would be to come up with a smaller loadup.el which
> only loads the stuff that is needed for temacs to start.  I didn't try
> that yet, but I did think that Phillip's work on ldefs-boot might just
> be a good starting point: those ldefs-boot-*.el files might be just
> what we need.

Oh, and more more thought: there could be a separate, smaller loadup
file for batch invocations, since speed of startup in that mode is
somewhat more important than in the interactive mode.

Re: Skipping unexec via a big .elc file

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Thu, 15 Dec 2016 21:59:08 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: emacs-devel@gnu.org
>> 
>> So my next idea would be to come up with a smaller loadup.el which
>> only loads the stuff that is needed for temacs to start.  I didn't try
>> that yet, but I did think that Phillip's work on ldefs-boot might just
>> be a good starting point: those ldefs-boot-*.el files might be just
>> what we need.
>
> Oh, and more more thought: there could be a separate, smaller loadup
> file for batch invocations, since speed of startup in that mode is
> somewhat more important than in the interactive mode.


loadup would need refactoring anyway. It doesn't just loadup at the
moment. It also dumps and kills Emacs. It's also got some strange
syntactic constraints because the Makefile does some sed based parsing
of it.

Phil

Re: Skipping unexec via a big .elc file

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
> So my next idea would be to come up with a smaller loadup.el which
> only loads the stuff that is needed for temacs to start.  I didn't try
> that yet, but I did think that Phillip's work on ldefs-boot might just
> be a good starting point: those ldefs-boot-*.el files might be just
> what we need.
>
> IMO, it would be interesting to see where this will take us, and what
> kind of performance could that produce.


I looked at this a little and in fact the boot code that I have written
does tell you exactly which autoloads you need to get temacs to work --
it's not very many, I think that there are only 10 or so (bytecomp.el
for instance).

Of course, this is 10 autoloads PLUS all of the non-auto loads in
loadup.el. My own feeling is that this is a bit unclean at the moment;
given that loadup.el is supposed to support temacs till the point that
it dumps, probably all of the autoloads used for this process should be
explicitly in loadup; or, alternatively, we should have very non-auto
loads in loadup and do everything via autoload.

Phil

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

On Dec 19, 2016, at 10:09, Phillip Lord <phillip.lord@russet.org.uk> wrote:

> I looked at this a little and in fact the boot code that I have written
> does tell you exactly which autoloads you need to get temacs to work --
> it's not very many, I think that there are only 10 or so (bytecomp.el
> for instance).

This sounds like it could be the biggest help for startup time at this point.  Are you going to look further into making a lightweight loadup file?

Looking at ldefs-boot.el and loaddefs.el, and contemplating the parsing of them, I wonder: If we go the big-elc route, can we defer loading the doc strings until they’re actually needed?  Perhaps using the “(#$ . nnnn)” syntax used in .elc files, or somehow pointing at the real .el or .elc files defining the functions?  Maybe just omit the function doc strings, if the help code does something reasonable in that case?

I’ve still been poking at the reader code, but for small-ish changes I think I’m hitting a point of diminishing returns.  My current test case run time is about 0.15-0.16s, though the run times are short enough that minor system activity at the same time can affect the results.  I’ve got one more experiment in the works that cuts almost 20% of the size of dumped.elc, and cuts the test run time to about 0.14s.  (Sharing interned symbols in the printer, so “setplist … setplist …” becomes “#4=setplist … #4# …”, drastically cutting into the 90% of oblookup calls that are done for symbols already in the obarray, and the related string manipulations, as well as the legibility of the generated file.)  After that, I think the next step is further specialization of read1/readchar/read_escape/readbyte for the get-file-char case, and maybe more tweaks to try to optimize for mostly-ASCII input.  But those result in more code duplication and additional maintenance work, for probably small benefit, so they’re not looking all that appealing.

So, at this point, I’m inclined to finish my current experiment with the printer, and maybe set aside work on the big-elc performance for a bit, maybe look into threading bugs or the state of the CANNOT_DUMP code.

Ken

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

> them, I wonder: If we go the big-elc route, can we defer loading the doc
> strings until they’re actually needed?  Perhaps using the “(#$ . nnnn)”

In the dumped.elc file I generate, there should be basically
no docstrings (the data I dump already uses either the NNN or the (#$
. NNN) representation to point to docstrings in the DOC file or in the
original .elc file), so I don't think there's much opportunity for deferral.


        Stefan

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

On Dec 20, 2016, at 18:22, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

>> them, I wonder: If we go the big-elc route, can we defer loading the doc
>> strings until they’re actually needed?  Perhaps using the “(#$ . nnnn)”
> 
> In the dumped.elc file I generate, there should be basically
> no docstrings (the data I dump already uses either the NNN or the (#$
> . NNN) representation to point to docstrings in the DOC file or in the
> original .elc file), so I don't think there's much opportunity for deferral.

Ah, yes, I forgot that happens even for the loaddefs doc strings not explicitly using that syntax, thanks to Snarf-documentation.  At least, so long as all the files we might pre-load under various conditions are all covered by the DOC file, if we do take the approach of a smaller file for batch mode.

Ken

Re: Skipping unexec via a big .elc file

[Phillip Lord]

Ken Raeburn <raeburn@raeburn.org> writes:

> On Dec 19, 2016, at 10:09, Phillip Lord <phillip.lord@russet.org.uk> wrote:
>
>> I looked at this a little and in fact the boot code that I have written
>> does tell you exactly which autoloads you need to get temacs to work --
>> it's not very many, I think that there are only 10 or so (bytecomp.el
>> for instance).
>
> This sounds like it could be the biggest help for startup time at this point.
> Are you going to look further into making a lightweight loadup file?
>
> Looking at ldefs-boot.el and loaddefs.el, and contemplating the parsing of
> them, I wonder: If we go the big-elc route, can we defer loading the doc
> strings until they’re actually needed?  Perhaps using the “(#$ . nnnn)” syntax
> used in .elc files, or somehow pointing at the real .el or .elc files defining
> the functions?  Maybe just omit the function doc strings, if the help code
> does something reasonable in that case?

In the ldefs-boot-auto.el file, there are no doc strings. This was
mostly because it was extra effort to add them, and they made no
difference for the use intended.

One simple way to lazy load doc strings would be to fiddle with help
code so that it loads the relevant file before looking for the
docstring. This does assume that the all the code in dumped.elc is
idempotent, though.

Incidentally, unless I have misunderstood, dumped.elc will duplicate
code also found in other .elc files (say, byte-run.elc, and
nadvice.elc)? Is dumped.elc going to detect these dependencies and
redump as necessary?

Phil

Re: Skipping unexec via a big .elc file

[Robert Pluim]

Ken Raeburn <raeburn@raeburn.org> writes:

> Branch scratch/raeburn-startup deleted and re-pushed.
>
> In addition to the changes I mentioned earlier, I found an unnecessary
> memset in the face reinitialization code that could go, and an
> initialization form was being emitted that tried to incorporate the
> obarray by value (which wouldn’t work because the symbol chains don’t
> all get dumped); omitting the latter for now cuts the file size a
> percent or so.

Hmm, it crashes for me when doing a bootstrap on GNU/Linux Mint (based
on Ubuntu 16.04)

/bin/bash: line 1: 21979 Aborted                 EMACSLOADPATH= '../src/emacs' -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -f batch-byte-compile calc/calcalg2.el
Makefile:282: recipe for target 'calc/calcalg2.elc' failed

Backtrace (this is compiled -ggdb -O0 with gcc 5.4.0)

(gdb) bt
#0  terminate_due_to_signal (sig=6, backtrace_limit=40) at emacs.c:367
#1  0x00000000005866d3 in emacs_abort () at sysdep.c:2337
#2  0x000000000057100d in unblock_input_to (level=-1) at keyboard.c:7170
#3  0x0000000000571024 in unblock_input () at keyboard.c:7186
#4  0x0000000000639605 in read1 (readcharfun=25152, pch=0x7fffffff719c, first_in_list=false) at lread.c:3446
#5  0x000000000063a906 in read_list (flag=false, readcharfun=25152) at lread.c:3928
#6  0x00000000006371e9 in read1 (readcharfun=25152, pch=0x7fffffff7514, first_in_list=false) at lread.c:2629
#7  0x0000000000636336 in read0 (readcharfun=25152) at lread.c:2210
#8  0x00000000006361fb in read_internal_start (stream=25152, start=0, end=0) at lread.c:2176
#9  0x0000000000635ef2 in Fread (stream=25152) at lread.c:2110
#10 0x000000000060748d in Ffuncall (nargs=2, args=0x7fffffff76b0) at eval.c:2715
#11 0x0000000000606d54 in call1 (fn=39792, arg1=25152) at eval.c:2574
#12 0x00000000006358a5 in readevalloop (readcharfun=25152, stream=0x19bc7a0, sourcename=26993220, printflag=false, 
    unibyte=0, readfun=0, start=0, end=0) at lread.c:1958
#13 0x0000000000633fed in Fload (file=10190276, noerror=0, nomessage=45216, nosuffix=0, must_suffix=45216)
    at lread.c:1367
#14 0x00000000006136d9 in Frequire (feature=4534752, filename=0, noerror=0) at fns.c:2894
#15 0x0000000000607516 in Ffuncall (nargs=2, args=0x7fffffff7ae8) at eval.c:2722
#16 0x000000000064e9ef in exec_byte_code (bytestr=26973012, vector=22861437, maxdepth=10, args_template=0, nargs=0, 
    args=0x0) at bytecode.c:639
#17 0x000000000064dd56 in Fbyte_code (bytestr=26973012, vector=22861437, maxdepth=10) at bytecode.c:319
#18 0x0000000000605f02 in eval_sub (form=26933523) at eval.c:2194
#19 0x00000000006359c3 in readevalloop (readcharfun=25152, stream=0x18c7190, sourcename=26972436, printflag=false, 
    unibyte=0, readfun=0, start=0, end=0) at lread.c:1980
#20 0x0000000000633fed in Fload (file=26802292, noerror=0, nomessage=45216, nosuffix=0, must_suffix=45216)
    at lread.c:1367
#21 0x00000000006136d9 in Frequire (feature=13579232, filename=0, noerror=0) at fns.c:2894
#22 0x0000000000607516 in Ffuncall (nargs=2, args=0x7fffffff8518) at eval.c:2722
#23 0x00000000006063fa in Fapply (nargs=2, args=0x7fffffff8518) at eval.c:2300
#24 0x000000000060735d in Ffuncall (nargs=3, args=0x7fffffff8510) at eval.c:2695
#25 0x000000000064e9ef in exec_byte_code (bytestr=25727860, vector=22453597, maxdepth=38, args_template=1030, 
    nargs=1, args=0x7fffffff8a78) at bytecode.c:639
#26 0x0000000000607dd8 in funcall_lambda (fun=22453765, nargs=1, arg_vector=0x7fffffff8a70) at eval.c:2879
#27 0x00000000006077ae in Ffuncall (nargs=2, args=0x7fffffff8a68) at eval.c:2764
#28 0x000000000064e9ef in exec_byte_code (bytestr=25715156, vector=22445469, maxdepth=18, args_template=1030, 
    nargs=1, args=0x7fffffff8f60) at bytecode.c:639
#29 0x0000000000607dd8 in funcall_lambda (fun=22445517, nargs=1, arg_vector=0x7fffffff8f58) at eval.c:2879
#30 0x00000000006077ae in Ffuncall (nargs=2, args=0x7fffffff8f50) at eval.c:2764
#31 0x000000000064e9ef in exec_byte_code (bytestr=25714756, vector=22445325, maxdepth=22, args_template=1030, 
    nargs=1, args=0x7fffffff9438) at bytecode.c:639
#32 0x0000000000607dd8 in funcall_lambda (fun=22445373, nargs=1, arg_vector=0x7fffffff9430) at eval.c:2879
#33 0x00000000006077ae in Ffuncall (nargs=2, args=0x7fffffff9428) at eval.c:2764
#34 0x000000000064e9ef in exec_byte_code (bytestr=17023828, vector=22255765, maxdepth=42, args_template=2058, 
    nargs=2, args=0x7fffffff9948) at bytecode.c:639
#35 0x0000000000607dd8 in funcall_lambda (fun=22255869, nargs=2, arg_vector=0x7fffffff9938) at eval.c:2879
#36 0x00000000006077ae in Ffuncall (nargs=3, args=0x7fffffff9930) at eval.c:2764
#37 0x000000000064e9ef in exec_byte_code (bytestr=25714692, vector=22433525, maxdepth=18, args_template=1030, 
    nargs=1, args=0x7fffffff9e08) at bytecode.c:639
#38 0x0000000000607dd8 in funcall_lambda (fun=22445421, nargs=1, arg_vector=0x7fffffff9e00) at eval.c:2879
#39 0x00000000006077ae in Ffuncall (nargs=2, args=0x7fffffff9df8) at eval.c:2764
#40 0x000000000064e9ef in exec_byte_code (bytestr=25699108, vector=22436957, maxdepth=22, args_template=1030, 
    nargs=1, args=0x7fffffffa330) at bytecode.c:639
#41 0x0000000000607dd8 in funcall_lambda (fun=22433429, nargs=1, arg_vector=0x7fffffffa328) at eval.c:2879
#42 0x00000000006077ae in Ffuncall (nargs=2, args=0x7fffffffa320) at eval.c:2764
#43 0x000000000064e9ef in exec_byte_code (bytestr=25698596, vector=22437133, maxdepth=66, args_template=1030, 
    nargs=1, args=0x7fffffffa978) at bytecode.c:639
#44 0x0000000000607dd8 in funcall_lambda (fun=22433477, nargs=1, arg_vector=0x7fffffffa970) at eval.c:2879
#45 0x00000000006077ae in Ffuncall (nargs=2, args=0x7fffffffa968) at eval.c:2764
#46 0x000000000064e9ef in exec_byte_code (bytestr=25677076, vector=22432581, maxdepth=66, args_template=2054, 
    nargs=1, args=0x7fffffffb030) at bytecode.c:639
#47 0x0000000000607dd8 in funcall_lambda (fun=22429357, nargs=1, arg_vector=0x7fffffffb028) at eval.c:2879
#48 0x00000000006077ae in Ffuncall (nargs=2, args=0x7fffffffb020) at eval.c:2764
#49 0x000000000064e9ef in exec_byte_code (bytestr=25918004, vector=19599701, maxdepth=34, args_template=1030, 
    nargs=1, args=0x7fffffffb578) at bytecode.c:639
#50 0x0000000000607dd8 in funcall_lambda (fun=19599829, nargs=1, arg_vector=0x7fffffffb570) at eval.c:2879
#51 0x00000000006077ae in Ffuncall (nargs=2, args=0x7fffffffb568) at eval.c:2764
#52 0x000000000064e9ef in exec_byte_code (bytestr=25917348, vector=20758333, maxdepth=42, args_template=1026, 
    nargs=0, args=0x7fffffffbb58) at bytecode.c:639
#53 0x0000000000607dd8 in funcall_lambda (fun=20758493, nargs=0, arg_vector=0x7fffffffbb58) at eval.c:2879
#54 0x00000000006077ae in Ffuncall (nargs=1, args=0x7fffffffbb50) at eval.c:2764
#55 0x000000000064e9ef in exec_byte_code (bytestr=10932540, vector=10932573, maxdepth=94, args_template=1030, 
    nargs=1, args=0x7fffffffc4e8) at bytecode.c:639
#56 0x0000000000607dd8 in funcall_lambda (fun=10932493, nargs=1, arg_vector=0x7fffffffc4e0) at eval.c:2879
#57 0x00000000006077ae in Ffuncall (nargs=2, args=0x7fffffffc4d8) at eval.c:2764
#58 0x000000000064e9ef in exec_byte_code (bytestr=10909516, vector=10909549, maxdepth=86, args_template=2, nargs=0, 
    args=0x7fffffffd108) at bytecode.c:639
#59 0x0000000000607dd8 in funcall_lambda (fun=10909469, nargs=0, arg_vector=0x7fffffffd108) at eval.c:2879
#60 0x00000000006077ae in Ffuncall (nargs=1, args=0x7fffffffd100) at eval.c:2764
#61 0x000000000064e9ef in exec_byte_code (bytestr=10905556, vector=10905589, maxdepth=50, args_template=2, nargs=0, 
    args=0x7fffffffd6f0) at bytecode.c:639
#62 0x0000000000607dd8 in funcall_lambda (fun=10905509, nargs=0, arg_vector=0x7fffffffd6f0) at eval.c:2879
#63 0x0000000000607b46 in apply_lambda (fun=10905509, args=0, count=4) at eval.c:2816
#64 0x000000000060607e in eval_sub (form=19109027) at eval.c:2233
#65 0x0000000000605572 in Feval (form=19109027, lexical=0) at eval.c:2010
#66 0x00000000005644f6 in top_level_2 () at keyboard.c:1127
#67 0x0000000000603fbd in internal_condition_case (bfun=0x5644d3 <top_level_2>, handlers=19536, 
    hfun=0x563f70 <cmd_error>) at eval.c:1314
#68 0x0000000000564537 in top_level_1 (ignore=0) at keyboard.c:1135
#69 0x00000000006038cc in internal_catch (tag=46608, func=0x5644f8 <top_level_1>, arg=0) at eval.c:1080
#70 0x000000000056442b in command_loop () at keyboard.c:1096
#71 0x0000000000563b55 in recursive_edit_1 () at keyboard.c:703
#72 0x0000000000563ccc in Frecursive_edit () at keyboard.c:774
#73 0x0000000000561951 in main (argc=9, argv=0x7fffffffdc38) at emacs.c:1698

Lisp Backtrace:
"read" (0xffff76b8)
"require" (0xffff7af0)
"byte-code" (0xffff7f20)
"require" (0xffff8520)
"apply" (0xffff8518)
"byte-compile-file-form-require" (0xffff8a70)
"byte-compile-file-form" (0xffff8f58)
0x1567d38 PVEC_COMPILED
"byte-compile-recurse-toplevel" (0xffff9938)
"byte-compile-toplevel-file-form" (0xffff9e00)
0x1564e90 PVEC_COMPILED
"byte-compile-from-buffer" (0xffffa970)
"byte-compile-file" (0xffffb028)
"batch-byte-compile-file" (0xffffb570)
"batch-byte-compile" (0xffffbb58)
"command-line-1" (0xffffc4e0)
"command-line" (0xffffd108)
"normal-top-level" (0xffffd6f0)
(gdb) 

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Sun, 11 Dec 2016 08:34:01 -0500
> 
> With all these changes — Stefan’s new patch with additional initialization, and my updates to shave a little more time off — I’m still hitting just under 0.2s for:
> 
>   time ./temacs --batch --eval '(progn (message "hi") (kill-emacs))'
> 
> on Linux/GNU/X11 (Intel Core i5-2320, 3GHz, gcc 4.9); my Mac (Intel Core 2 Duo, 2.8GHz) takes over half a second (including at least one GC invocation).

For the record, my timing is 0.828s with an unoptimized build of the
branch, as opposed to 13.2s with an unoptimized build on master, and
5.343s with an optimized (-Og) build of Emacs 25.1.90.  The CPU is
Core i7-2600, 3.4GHz; the compiler used is GCC 5.3.0.

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> Date: Sat, 24 Dec 2016 15:37:11 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> I’m still hitting just under 0.2s for:
> 
> >   time ./temacs --batch --eval '(progn (message "hi") (kill-emacs))'
> > 
> > on Linux/GNU/X11 (Intel Core i5-2320, 3GHz, gcc 4.9); my Mac (Intel Core 2 Duo, 2.8GHz) takes over half a second (including at least one GC invocation).
> 
> For the record, my timing is 0.828s with an unoptimized build of the
> branch, as opposed to 13.2s with an unoptimized build on master, and
> 5.343s with an optimized (-Og) build of Emacs 25.1.90.  The CPU is
> Core i7-2600, 3.4GHz; the compiler used is GCC 5.3.0.

With an optimized (-O2) build on the same system, the above command
takes 0.190s on the average.  Byte-compiling Lisp files in batch mode
shows that loading dumped.elc takes about 0.150s, as that is the
difference between byte-compiling with temacs and the dumped emacs.
For example, compiling simple.el takes 0.656s with temacs and 0.515s
with dumped emacs.  IOW, the overhead is additive, as expected.

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

Ken,

I tried to get rid of calling dump-emacs in the raeburn-startup
branch, see the changes below.  The resulting code builds and produces
dumped.elc, but then fails to compile the *.el files:

  ...
  Loading d:/gnu/git/emacs/no-unexec/lisp/leim/leim-list.el (source)...
  Finding pointers to doc strings...
  Finding pointers to doc strings...done
  Dumping under the name emacs
  Dumping into dumped.elc...preparing...
  Dumping into dumped.elc...generating...
  Dumping into dumped.elc...printing...
  Dumping into dumped.elc...saving...
  Dumping into dumped.elc...done
  mv -f emacs.exe bootstrap-emacs.exe
  make -C ../lisp compile-first EMACS="../src/bootstrap-emacs.exe"
  make[2]: Entering directory `/d/gnu/git/emacs/no-unexec/lisp'
    ELC      emacs-lisp/macroexp.elc
  Loading ../src/dumped.elc...
  Multiple args to , are not supported: ((\, (quote set-window-parameter)) temp (\, (quote set-window-parameter)) end)
    ELC      emacs-lisp/cconv.elc
  Loading ../src/dumped.elc...
  Multiple args to , are not supported: ((\, (quote set-window-parameter)) temp (\, (quote set-window-parameter)) end)

This could be related to the fact that the original code produced the
first dumped.elc in the top-level directory, not in src/, and I needed
to fix that, since otherwise bootstrap-emacs would exit immediately
(see the changes below).  In the original version, src/dumped.elc was
only produced after all the necessary Lisp files were byte-compiled
already.

So it seems like the current build process on this branch still
somehow depends on a dumped emacs executable, until it byte-compiles
all the preloaded Lisp files, and produces dumped.elc from that.  IOW,
the first dumped.elc produced before byte-compiling those files is not
up to the job of running Emacs for byte-compiling Lisp files.  How can
we fix that, so that unexec and its call can be really removed from
the sources?  Or did I miss something?

Thanks.

diff --git a/lisp/loadup.el b/lisp/loadup.el
index 54d19c1..873d804 100644
--- a/lisp/loadup.el
+++ b/lisp/loadup.el
@@ -453,27 +453,30 @@
       ;; confused people installing Emacs (they'd install the file
       ;; under the name `xemacs'), and it's inconsistent with every
       ;; other GNU program's build process.
-      (dump-emacs "emacs" "temacs")
-      (message "%d pure bytes used" pure-bytes-used)
-      ;; Recompute NAME now, so that it isn't set when we dump.
-      (if (not (or (eq system-type 'ms-dos)
-                   ;; Don't bother adding another name if we're just
-                   ;; building bootstrap-emacs.
-                   (equal (last command-line-args) '("bootstrap"))))
-	  (let ((name (concat "emacs-" emacs-version))
-		(exe (if (eq system-type 'windows-nt) ".exe" "")))
-	    (while (string-match "[^-+_.a-zA-Z0-9]+" name)
-	      (setq name (concat (downcase (substring name 0 (match-beginning 0)))
+      ;; (dump-emacs "emacs" "temacs")
+      ;; (message "%d pure bytes used" pure-bytes-used)
+      (let ((exe (if (memq system-type '(windows-nt ms-dos)) ".exe" "")))
+        (copy-file (expand-file-name (concat "temacs" exe) invocation-directory)
+                   (expand-file-name (concat "emacs" exe) invocation-directory)
+                   t)
+        ;; Recompute NAME now, so that it isn't set when we dump.
+        (if (not (or (eq system-type 'ms-dos)
+                     ;; Don't bother adding another name if we're just
+                     ;; building bootstrap-emacs.
+                     (equal (last command-line-args) '("bootstrap"))))
+            (let ((name (concat "emacs-" emacs-version)))
+              (while (string-match "[^-+_.a-zA-Z0-9]+" name)
+                (setq name (concat (downcase (substring name 0 (match-beginning 0)))
 				 "-"
 				 (substring name (match-end 0)))))
-	    (setq name (concat name exe))
-            (message "Adding name %s" name)
-	    ;; When this runs on Windows, invocation-directory is not
-	    ;; necessarily the current directory.
-	    (add-name-to-file (expand-file-name (concat "emacs" exe)
-						invocation-directory)
-			      (expand-file-name name invocation-directory)
-			      t)))
+              (setq name (concat name exe))
+              (message "Adding name %s" name)
+              ;; When this runs on Windows, invocation-directory is not
+              ;; necessarily the current directory.
+              (add-name-to-file (expand-file-name (concat "emacs" exe)
+                                                  invocation-directory)
+                                (expand-file-name name invocation-directory)
+                                t))))
       (message "Dumping into dumped.elc...preparing...")
 
       ;; Dump the current state into a file so we can reload it!
@@ -555,6 +558,7 @@
          obarray)
         (message "Dumping into dumped.elc...printing...")
         (with-current-buffer (generate-new-buffer "dumped.elc")
+          (setq default-directory invocation-directory)
           (insert ";ELC\^W\^@\^@\^@\n;;; Compiled\n;;; in Emacs version "
                   emacs-version "\n")
           (let ((print-circle t)

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

> On Jan 7, 2017, at 04:40, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> Ken,
> 
> I tried to get rid of calling dump-emacs in the raeburn-startup
> branch, see the changes below.  The resulting code builds and produces
> dumped.elc, but then fails to compile the *.el files:

I’ve been looking into it this weekend.  It appears that in some of my builds I’m seeing in dumped.elc stuff along the lines of:

(setplist 'window-parameter '(gv-expander (closure (t) #19=(do &rest args) (gv--defsetter 'window-parameter (lambda #20=(val &rest args) `(,'set-window-parameter . #21=(,@args ,val))) . #22=(do args))) side-effect-free t))

That’s with my #N# patch removed; that patch obfuscates the code but I don’t think it should be changing the meaning.

The comma-quote-symbol syntax looks strange to me, could that be causing it?

> This could be related to the fact that the original code produced the
> first dumped.elc in the top-level directory, not in src/, and I needed
> to fix that, since otherwise bootstrap-emacs would exit immediately
> (see the changes below).  In the original version, src/dumped.elc was
> only produced after all the necessary Lisp files were byte-compiled
> already.

In the GNU/Linux build, the dumped.elc file is generated in the src directory of the build tree.  So that part of your patch didn’t alter anything for my testing as far as I can see.

But the GNU/Linux build supports building in a separate tree from the source tree, a mode I usually do my builds in, and at startup we look for dumped.elc in the src directory of the source tree, not the build tree.  So I still have to tweak it manually.

> So it seems like the current build process on this branch still
> somehow depends on a dumped emacs executable, until it byte-compiles
> all the preloaded Lisp files, and produces dumped.elc from that.  IOW,
> the first dumped.elc produced before byte-compiling those files is not
> up to the job of running Emacs for byte-compiling Lisp files.  How can
> we fix that, so that unexec and its call can be really removed from
> the sources?  Or did I miss something?

A workaround might be to use loadup.el instead of dumped.elc during that stage.  But that doesn’t fix the problem.

Ken

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

> `(,'set-window-parameter . #21=(,@args ,val))) . #22=(do args)))
> side-effect-free t))
>
> The comma-quote-symbol syntax looks strange to me, could that be causing it?

The ,' is a result of evaluation of code like

    ``(,',setter ,@args ,val)

so, it's indeed strange, but only to the extent that nested backquotes
are "strange".

Eli wrote:
>  Multiple args to , are not supported: ((\, (quote set-window-parameter)) temp (\, (quote set-window-parameter)) end)

Hmm... I don't understand this.  This message seems to come from
backquote.el:

   ((eq (car s) backquote-unquote-symbol)
    (if (<= level 0)
        (cond
         ((> (length s) 2)
          ;; We could support it with: (cons 2 `(list . ,(cdr s)))
          ;; But let's not encourage such uses.
          (error "Multiple args to , are not supported: %S" s))
         (t (cons (if (eq (car-safe (nth 1 s)) 'quote) 0 1)
                  (nth 1 s))))
      (backquote-delay-process s (1- level))))

but then `s` should have \, in its car, whereas the above message
indicates that (car s) is (\, (quote set-window-parameter)) which
implies we should not have entered this branch.

Maybe I'm just too tired to read this code right, tho.


        Stefan

Re: Skipping unexec via a big .elc file

[Andreas Schwab]

On Jan 09 2017, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> Eli wrote:
>>  Multiple args to , are not supported: ((\, (quote set-window-parameter)) temp (\, (quote set-window-parameter)) end)
>
> Hmm... I don't understand this.  This message seems to come from
> backquote.el:
>
>    ((eq (car s) backquote-unquote-symbol)
>     (if (<= level 0)
>         (cond
>          ((> (length s) 2)
>           ;; We could support it with: (cons 2 `(list . ,(cdr s)))
>           ;; But let's not encourage such uses.
>           (error "Multiple args to , are not supported: %S" s))
>          (t (cons (if (eq (car-safe (nth 1 s)) 'quote) 0 1)
>                   (nth 1 s))))
>       (backquote-delay-process s (1- level))))
>
> but then `s` should have \, in its car, whereas the above message
> indicates that (car s) is (\, (quote set-window-parameter)) which
> implies we should not have entered this branch.

That can only mean that something clobbered backquote-unquote-symbol.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Tue, 10 Jan 2017 10:46:25 +0100
> Cc: emacs-devel@gnu.org
> 
> On Jan 09 2017, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
> > Eli wrote:
> >>  Multiple args to , are not supported: ((\, (quote set-window-parameter)) temp (\, (quote set-window-parameter)) end)
> >
> > Hmm... I don't understand this.  This message seems to come from
> > backquote.el:
> >
> >    ((eq (car s) backquote-unquote-symbol)
> >     (if (<= level 0)
> >         (cond
> >          ((> (length s) 2)
> >           ;; We could support it with: (cons 2 `(list . ,(cdr s)))
> >           ;; But let's not encourage such uses.
> >           (error "Multiple args to , are not supported: %S" s))
> >          (t (cons (if (eq (car-safe (nth 1 s)) 'quote) 0 1)
> >                   (nth 1 s))))
> >       (backquote-delay-process s (1- level))))
> >
> > but then `s` should have \, in its car, whereas the above message
> > indicates that (car s) is (\, (quote set-window-parameter)) which
> > implies we should not have entered this branch.
> 
> That can only mean that something clobbered backquote-unquote-symbol.

Yes, the value of backquote-unquote-symbol at this point is indeed
this:

   (\, (quote set-window-parameter))

I guess something is wrong with reading dumped.elc?

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

> On Jan 10, 2017, at 12:19, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Date: Tue, 10 Jan 2017 10:46:25 +0100
>> Cc: emacs-devel@gnu.org
>> 
>> On Jan 09 2017, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> 
>>> Eli wrote:
>>>> Multiple args to , are not supported: ((\, (quote set-window-parameter)) temp (\, (quote set-window-parameter)) end)
>>> 
>>> Hmm... I don't understand this.  This message seems to come from
>>> backquote.el:
>>> 
>>>   ((eq (car s) backquote-unquote-symbol)
>>>    (if (<= level 0)
>>>        (cond
>>>         ((> (length s) 2)
>>>          ;; We could support it with: (cons 2 `(list . ,(cdr s)))
>>>          ;; But let's not encourage such uses.
>>>          (error "Multiple args to , are not supported: %S" s))
>>>         (t (cons (if (eq (car-safe (nth 1 s)) 'quote) 0 1)
>>>                  (nth 1 s))))
>>>      (backquote-delay-process s (1- level))))
>>> 
>>> but then `s` should have \, in its car, whereas the above message
>>> indicates that (car s) is (\, (quote set-window-parameter)) which
>>> implies we should not have entered this branch.
>> 
>> That can only mean that something clobbered backquote-unquote-symbol.
> 
> Yes, the value of backquote-unquote-symbol at this point is indeed
> this:
> 
>   (\, (quote set-window-parameter))
> 
> I guess something is wrong with reading dumped.elc?

At the moment it’s looking to me like it might be a problem with my #N# patch for writing out symbols.  It got a little more of a speedup reading dumped.elc, but if I drop that change, I get a lot further in trying to bootstrap the tree with your change.  It still fails while processing the “leim” directory, though.

Indeed, looking at dumped.elc, I see:
  (#35# '#5646# '#218#)
where 35 is set-default, 5646 is backquote-unquote-symbol, and 218 is ,’set-window-parameter thanks to "#218=,’#897=set-window-parameter" being read from dumped.elc.  I suspect 218 was supposed to be just the comma, but the special printing of comma forms was still applied but is not compatible with the #N# handling, so comma and related symbols should just be excluded from that hack.

I’ll test that out, but in the meantime, commenting out the binding in loadup.el of print-symbols-as-references should make things work again (bootstrapping up until partway through the leim directory).

Ken

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

On Jan 11, 2017, at 01:32, Ken Raeburn <raeburn@raeburn.org> wrote:
> Indeed, looking at dumped.elc, I see:
>  (#35# '#5646# '#218#)
> where 35 is set-default, 5646 is backquote-unquote-symbol, and 218 is ,’set-window-parameter thanks to "#218=,’#897=set-window-parameter" being read from dumped.elc.  I suspect 218 was supposed to be just the comma, but the special printing of comma forms was still applied but is not compatible with the #N# handling, so comma and related symbols should just be excluded from that hack.

There were other instances that made it clear that “#218#” was being printed where “,” was intended, including with the lack of space before whatever followed that’s normal for a comma (e.g., “#218##219#” where #219# referred to some ordinary symbol).

I’ve just uploaded a workaround for that (including a comma-dot sequence that I’m not familiar with, but which seems to get the same treatment as comma and comma-at), and a bug fix I found relating to one of my earlier changes.

Now, with your patch to avoid unexec, it’s successfully compiling in the lisp directory but fails in leim, which I haven’t dug into yet:

make[2]: Entering directory '/home/raeburn/dev/emacs/s/lisp'
make -C ../leim all EMACS="../src/emacs"
make[3]: Entering directory '/home/raeburn/dev/emacs/s/leim'
/bin/mkdir -p ../lisp/leim/ja-dic
  GEN      ../lisp/leim/ja-dic/ja-dic.el
Loading ../src/dumped.elc...
Reading file "/home/raeburn/dev/emacs/s/leim/SKK-DIC/SKK-JISYO.L" ...
Processing OKURI-ARI entries ...
Debugger entered--Lisp error: (search-failed "^\\cH")
  re-search-forward("^\\cH")
  (let ((from (point)) to) (search-forward ";; okuri-nasi") (beginning-of-line) (setq to (point)) (narrow-to-region from to) (skkdic-convert-okuri-ari skkbuf buf) (widen) (goto-char to) (forward-line 1) (setq from (point)) (re-search-forward "^\\cH") (setq to (match-beginning 0)) (narrow-to-region from to) (skkdic-convert-postfix skkbuf buf) (widen) (goto-char to) (skkdic-convert-prefix skkbuf buf) (skkdic-collect-okuri-nasi) (skkdic-convert-okuri-nasi skkbuf buf) (save-current-buffer (set-buffer buf) (goto-char (point-max)) (insert ";;\n(provide 'ja-dic)\n\n" ";; Local Variables:\n" ";; version-control: never\n" ";; no-update-autoloads: t\n" ";; coding: utf-8\n" ";; End:\n\n" ";;; ja-dic.el ends here\n")))
…

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Thu, 12 Jan 2017 03:17:40 -0500
> Cc: Andreas Schwab <schwab@linux-m68k.org>,
>  monnier@iro.umontreal.ca,
>  emacs-devel@gnu.org
> 
> I’ve just uploaded a workaround for that (including a comma-dot sequence that I’m not familiar with, but which seems to get the same treatment as comma and comma-at), and a bug fix I found relating to one of my earlier changes.
> 
> Now, with your patch to avoid unexec, it’s successfully compiling in the lisp directory but fails

It does fail for me while byte compiling 2 files:

    ELC      leim/ja-dic/ja-dic.elc
  Loading ../src/dumped.elc...

  In toplevel form:
  leim/ja-dic/ja-dic.el:76:1:Error: Args out of range: " s  ", 2432, 2432
  Makefile:282: recipe for target `leim/ja-dic/ja-dic.elc' failed
  make[2]: *** [leim/ja-dic/ja-dic.elc] Error 1

    ELC      net/eww.elc
  Loading ../src/dumped.elc...

  In toplevel form:
  net/eww.el:29:1:Error: Undefined category: >
  Makefile:282: recipe for target `net/eww.elc' failed
  make[2]: *** [net/eww.elc] Error 1

The line number in the error message is bogus, it points to a require
line (that's a known issue with byte-compiler error reporting, I
think).  Running the latter compilation under GDB, I see this:

  Thread 1 hit Breakpoint 3, Fsignal (error_symbol=21616,
      data=-4611686018320478144) at eval.c:1471
  1471      signal_or_quit (error_symbol, data, false);
  (gdb) pp error_symbol
  error
  (gdb) pp data
  ("Undefined category: >")
  (gdb) bt
  #0  Fsignal (error_symbol=21616, data=-4611686018320478144) at eval.c:1471
  #1  0x0114c616 in xsignal (error_symbol=21616, data=-4611686018320478144)
      at lisp.h:3872
  #2  0x01221129 in xsignal1 (error_symbol=21616, arg=-9223372036747839088)
      at eval.c:1606
  #3  0x01221df8 in verror (
      m=0x167d0c5 <DEFAULT_REHASH_SIZE+373> "Undefined category: %c",
      ap=0x826bb4 ">") at eval.c:1791
  #4  0x01221e16 in error (
      m=0x167d0c5 <DEFAULT_REHASH_SIZE+373> "Undefined category: %c")
      at eval.c:1803
  #5  0x011108b2 in Fmodify_category_entry (character=4611686018427387937,
      category=4611686018427387966, table=-6917529027624565016, reset=0)
      at category.c:368
  #6  0x01225980 in Ffuncall (nargs=3, args=0x826d88) at eval.c:2726
  #7  0x0128959c in exec_byte_code (bytestr=-9223372036747840008,
      vector=-6917529027534179736, maxdepth=4611686018427387911,
      args_template=0, nargs=0, args=0x0) at bytecode.c:639
  #8  0x012886b0 in Fbyte_code (bytestr=-9223372036747840008,
      vector=-6917529027534179736, maxdepth=4611686018427387911)
      at bytecode.c:319
  #9  0x012238ef in eval_sub (form=-4611686018320476528) at eval.c:2194
  #10 0x01269693 in readevalloop (readcharfun=28056,
      stream=0x77c5fd00 <msvcrt!_iob+128>, sourcename=-9223372036747840088,
      printflag=false, unibyte=0, readfun=0, start=0, end=0) at lread.c:1980
  #11 0x01266f73 in Fload (file=-9223372036747853616, noerror=0,
      nomessage=55328, nosuffix=0, must_suffix=0) at lread.c:1367
  #12 0x01223a45 in eval_sub (form=-4611686018320474720) at eval.c:2202
  #13 0x0121c562 in Fprogn (body=-4611686018320474624) at eval.c:432
  #14 0x0121c277 in Fif (args=-4611686018320475808) at eval.c:390
  #15 0x012232ef in eval_sub (form=-4611686018320475824) at eval.c:2141
  #16 0x01268a56 in readevalloop_eager_expand_eval (val=-4611686018320474576,
      macroexpand=62241920) at lread.c:1792
  #17 0x01269679 in readevalloop (readcharfun=-6917529027540611248, stream=0x0,
      sourcename=-9223372036754301328, printflag=false, unibyte=0, readfun=0,
      start=0, end=0) at lread.c:1978
  #18 0x01269b55 in Feval_buffer (buffer=-6917529027540611248, printflag=0,
      filename=-9223372036754305384, unibyte=0, do_allow_print=55328)
      at lread.c:2044
  #19 0x01225a39 in Ffuncall (nargs=6, args=0x828098) at eval.c:2731
  #20 0x0128959c in exec_byte_code (bytestr=-9223372036760655744,
      vector=-6917529027547213384, maxdepth=4611686018427387910,
      args_template=0, nargs=0, args=0x0) at bytecode.c:639
  #21 0x01226fc8 in funcall_lambda (fun=-6917529027547213024, nargs=4,
      arg_vector=0x8286c0) at eval.c:2957
  #22 0x01225e1f in Ffuncall (nargs=5, args=0x8286b8) at eval.c:2764
  #23 0x01225110 in call4 (fn=64816760, arg1=-9223372036754305384,
      arg2=-9223372036754305384, arg3=0, arg4=55328) at eval.c:2599
  #24 0x01266b7c in Fload (file=-9223372036754314936, noerror=0,
      nomessage=55328, nosuffix=0, must_suffix=55328) at lread.c:1311
  #25 0x012391ce in Frequire (feature=76025368, filename=0, noerror=0)
      at fns.c:2894
  #26 0x012258f2 in Ffuncall (nargs=2, args=0x829068) at eval.c:2722
  #27 0x0122429d in Fapply (nargs=2, args=0x829068) at eval.c:2300
  #28 0x012256f3 in Ffuncall (nargs=3, args=0x829060) at eval.c:2695
  #29 0x0128959c in exec_byte_code (bytestr=-9223372036754937720,
      vector=-6917529027541241920, maxdepth=4611686018427387913,
      args_template=4611686018427388161, nargs=1, args=0x829648)
      at bytecode.c:639
  #30 0x012268de in funcall_lambda (fun=-6917529027541241752, nargs=1,
      arg_vector=0x829640) at eval.c:2879
  #31 0x01225e1f in Ffuncall (nargs=2, args=0x829638) at eval.c:2764
  #32 0x0128959c in exec_byte_code (bytestr=-9223372036754938152,
      vector=-6917529027541242792, maxdepth=4611686018427387908,
      args_template=4611686018427388161, nargs=1, args=0x829bb0)
      at bytecode.c:639
  #33 0x012268de in funcall_lambda (fun=-6917529027541242744, nargs=1,
      arg_vector=0x829ba8) at eval.c:2879
  #34 0x01225e1f in Ffuncall (nargs=2, args=0x829ba0) at eval.c:2764
  #35 0x0128959c in exec_byte_code (bytestr=-9223372036754938216,
      vector=-6917529027541242960, maxdepth=4611686018427387909,
      args_template=4611686018427388161, nargs=1, args=0x82a108)
      at bytecode.c:639
  #36 0x012268de in funcall_lambda (fun=-6917529027541242912, nargs=1,
      arg_vector=0x82a100) at eval.c:2879
  #37 0x01225e1f in Ffuncall (nargs=2, args=0x82a0f8) at eval.c:2764
  #38 0x0128959c in exec_byte_code (bytestr=-9223372036755022888,
      vector=-6917529027541352232, maxdepth=4611686018427387914,
      args_template=4611686018427388418, nargs=2, args=0x82a698)
      at bytecode.c:639
  #39 0x012268de in funcall_lambda (fun=-6917529027541352128, nargs=2,
      arg_vector=0x82a688) at eval.c:2879
  #40 0x01225e1f in Ffuncall (nargs=3, args=0x82a680) at eval.c:2764
  #41 0x0128959c in exec_byte_code (bytestr=-9223372036754938232,
      vector=-6917529027541242864, maxdepth=4611686018427387908,
      args_template=4611686018427388161, nargs=1, args=0x82abd8)
      at bytecode.c:639
  #42 0x012268de in funcall_lambda (fun=-6917529027541242840, nargs=1,
      arg_vector=0x82abd0) at eval.c:2879
  #43 0x01225e1f in Ffuncall (nargs=2, args=0x82abc8) at eval.c:2764
  #44 0x0128959c in exec_byte_code (bytestr=-9223372036754939440,
      vector=-6917529027541248760, maxdepth=4611686018427387909,
      args_template=4611686018427388161, nargs=1, args=0x82b180)
      at bytecode.c:639
  #45 0x012268de in funcall_lambda (fun=-6917529027541248584, nargs=1,
      arg_vector=0x82b178) at eval.c:2879
  #46 0x01225e1f in Ffuncall (nargs=2, args=0x82b170) at eval.c:2764
  #47 0x0128959c in exec_byte_code (bytestr=-9223372036754939488,
      vector=-6917529027541248536, maxdepth=4611686018427387920,
      args_template=4611686018427388161, nargs=1, args=0x82b848)
      at bytecode.c:639
  #48 0x012268de in funcall_lambda (fun=-6917529027541248088, nargs=1,
      arg_vector=0x82b840) at eval.c:2879
  #49 0x01225e1f in Ffuncall (nargs=2, args=0x82b838) at eval.c:2764
  #50 0x0128959c in exec_byte_code (bytestr=-9223372036754945616,
      vector=-6917529027541250008, maxdepth=4611686018427387920,
      args_template=4611686018427388417, nargs=1, args=0x82bf80)
      at bytecode.c:639
  #51 0x012268de in funcall_lambda (fun=-6917529027541249216, nargs=1,
      arg_vector=0x82bf78) at eval.c:2879
  #52 0x01225e1f in Ffuncall (nargs=2, args=0x82bf70) at eval.c:2764
  #53 0x0128959c in exec_byte_code (bytestr=-9223372036754832632,
      vector=-6917529027541134088, maxdepth=4611686018427387912,
      args_template=4611686018427388161, nargs=1, args=0x82c548)
      at bytecode.c:639
  #54 0x012268de in funcall_lambda (fun=-6917529027541133960, nargs=1,
      arg_vector=0x82c540) at eval.c:2879
  #55 0x01225e1f in Ffuncall (nargs=2, args=0x82c538) at eval.c:2764
  #56 0x0128959c in exec_byte_code (bytestr=-9223372036754832680,
      vector=-6917529027541140920, maxdepth=4611686018427387914,
      args_template=4611686018427388160, nargs=0, args=0x82cba8)
      at bytecode.c:639
  #57 0x012268de in funcall_lambda (fun=-6917529027541140760, nargs=0,
      arg_vector=0x82cba8) at eval.c:2879
  #58 0x01225e1f in Ffuncall (nargs=1, args=0x82cba0) at eval.c:2764
  #59 0x0128959c in exec_byte_code (bytestr=-9223372036757936560,
      vector=-6917529027544929360, maxdepth=4611686018427387927,
      args_template=4611686018427388161, nargs=1, args=0x82d5b8)
      at bytecode.c:639
  #60 0x012268de in funcall_lambda (fun=-6917529027544928504, nargs=1,
      arg_vector=0x82d5b0) at eval.c:2879
  #61 0x01225e1f in Ffuncall (nargs=2, args=0x82d5a8) at eval.c:2764
  #62 0x0128959c in exec_byte_code (bytestr=-9223372036760992488,
      vector=-6917529027547292792, maxdepth=4611686018427387925,
      args_template=4611686018427387904, nargs=0, args=0x82e258)
      at bytecode.c:639
  #63 0x012268de in funcall_lambda (fun=-6917529027547291064, nargs=0,
      arg_vector=0x82e258) at eval.c:2879
  #64 0x01225e1f in Ffuncall (nargs=1, args=0x82e250) at eval.c:2764
  #65 0x0128959c in exec_byte_code (bytestr=-9223372036756613024,
      vector=-6917529027542917320, maxdepth=4611686018427387916,
      args_template=4611686018427387904, nargs=0, args=0x82e850)
      at bytecode.c:639
  #66 0x012268de in funcall_lambda (fun=-6917529027542916696, nargs=0,
      arg_vector=0x82e850) at eval.c:2879
  #67 0x012263b2 in apply_lambda (fun=-6917529027542916696, args=0, count=21)
      at eval.c:2816
  #68 0x01223de3 in eval_sub (form=-4611686018332481616) at eval.c:2233
  #69 0x01222b56 in Feval (form=-4611686018332481616, lexical=0) at eval.c:2010
  #70 0x01223886 in eval_sub (form=-4611686018328424112) at eval.c:2191
  #71 0x0121c562 in Fprogn (body=-4611686018328424080) at eval.c:432
  #72 0x012232ef in eval_sub (form=-4611686018328424240) at eval.c:2141
  #73 0x01269693 in readevalloop (readcharfun=28056,
      stream=0x77c5fce0 <msvcrt!_iob+96>, sourcename=-9223372036838132000,
      printflag=false, unibyte=0, readfun=0, start=0, end=0) at lread.c:1980
  #74 0x01266f73 in Fload (file=-9223372036838132176, noerror=0, nomessage=0,
      nosuffix=0, must_suffix=0) at lread.c:1367
  #75 0x01223a45 in eval_sub (form=-4611686018340754848) at eval.c:2202
  #76 0x012205e2 in internal_lisp_condition_case (var=0,
      bodyform=-4611686018340754848, handlers=-4611686018340754768)
      at eval.c:1285
  #77 0x0121fe64 in Fcondition_case (args=-4611686018340754736) at eval.c:1211
  #78 0x012232ef in eval_sub (form=-4611686018340754720) at eval.c:2141
  #79 0x01222b56 in Feval (form=-4611686018340754720, lexical=0) at eval.c:2010
  #80 0x01155640 in top_level_2 () at keyboard.c:1127
  #81 0x01220675 in internal_condition_case (bfun=0x115560a <top_level_2>,
      handlers=21616, hfun=0x1154dc1 <cmd_error>) at eval.c:1314
  #82 0x011556a6 in top_level_1 (ignore=0) at keyboard.c:1135
  #83 0x0121f7fd in internal_catch (tag=57512, func=0x1155646 <top_level_1>,
      arg=0) at eval.c:1080
  #84 0x01155522 in command_loop () at keyboard.c:1096
  #85 0x011547f3 in recursive_edit_1 () at keyboard.c:703
  #86 0x01154a8f in Frecursive_edit () at keyboard.c:774
  #87 0x01152244 in main (argc=7, argv=0xa440d8) at emacs.c:1698

  Lisp Backtrace:
  "modify-category-entry" (0x826d90)
  "byte-code" (0x827270)
  "load" (0x827a90)
  "if" (0x827cc0)
  "eval-buffer" (0x8280a0)
  "load-with-code-conversion" (0x8286c0)
  "require" (0x829070)
  "apply" (0x829068)
  "byte-compile-file-form-require" (0x829640)
  "byte-compile-file-form" (0x829ba8)
  0x5f36be0 PVEC_COMPILED
  "byte-compile-recurse-toplevel" (0x82a688)
  "byte-compile-toplevel-file-form" (0x82abd0)
  0x5f355b8 PVEC_COMPILED
  "byte-compile-from-buffer" (0x82b840)
  "byte-compile-file" (0x82bf78)
  "batch-byte-compile-file" (0x82c540)
  "batch-byte-compile" (0x82cba8)
  "command-line-1" (0x82d5b0)
  "command-line" (0x82e258)
  "normal-top-level" (0x82e850)
  "eval" (0x82eb30)
  "progn" (0x82ed20)
  "load" (0x82f500)
  "condition-case" (0x82f7d0)
  (gdb) fr 11
  #11 0x01266f73 in Fload (file=-9223372036747853616, noerror=0,
      nomessage=55328, nosuffix=0, must_suffix=0) at lread.c:1367
  1367        readevalloop (Qget_file_char, stream, hist_file_name,
  (gdb) pp file
  "kinsoku"
  (gdb)

So it is loading kinsoku.el, and the code which triggers this is this:

  (while (< idx len)
    (setq ch (aref kinsoku-bol idx)
	  idx (1+ idx))
    (modify-category-entry ch ?>)))

The category '>' is defined in characters.el.  Surprisingly,
characters.elc in this branch is identical to the file on master, so
byte compilation (see below) is off the hook here.  What else could
explain that this category is deemed unknown?

Running the ja-dic.el compilation under GDB, I see this:

  Thread 1 hit Breakpoint 3, Fsignal (error_symbol=9464,
      data=-4611686018325493760) at eval.c:1471
  1471      signal_or_quit (error_symbol, data, false);
  args-out-of-range
  (gdb) pp data
  (" s  " 2432 2432)
  (gdb) bt
  #0  Fsignal (error_symbol=9464, data=-4611686018325493760) at eval.c:1471
  #1  0x0114c616 in xsignal (error_symbol=9464, data=-4611686018325493760)
      at lisp.h:3872
  #2  0x0122120b in xsignal3 (error_symbol=9464, arg1=-9223372036754289936,
      arg2=4611686018427390336, arg3=4611686018427390336) at eval.c:1618
  #3  0x011f8170 in args_out_of_range_3 (a1=-9223372036754289936,
      a2=4611686018427390336, a3=4611686018427390336) at data.c:169
  #4  0x0122f302 in validate_subarray (array=-9223372036754289936,
      from=4611686018427390336, to=4611686018427390336, size=4, ifrom=0x828da8,
      ito=0x828da4) at fns.c:1257
  #5  0x0122f3a1 in Fsubstring (string=-9223372036754289936,
      from=4611686018427390336, to=4611686018427390336) at fns.c:1282
  #6  0x0128a8b7 in exec_byte_code (bytestr=-9223372036754295792,
      vector=-6917529027540600688, maxdepth=4611686018427387908,
      args_template=0, nargs=0, args=0x0) at bytecode.c:958
  #7  0x01226fc8 in funcall_lambda (fun=-6917529027540652976, nargs=1,
      arg_vector=0x829398) at eval.c:2957
  #8  0x01225e1f in Ffuncall (nargs=2, args=0x829390) at eval.c:2764
  #9  0x0128959c in exec_byte_code (bytestr=-9223372036754295728,
      vector=-6917529027540600592, maxdepth=4611686018427387912,
      args_template=0, nargs=0, args=0x0) at bytecode.c:639
  #10 0x01226fc8 in funcall_lambda (fun=-6917529027540600496, nargs=15663,
      arg_vector=0x60c72d0) at eval.c:2957
  #11 0x01225e1f in Ffuncall (nargs=15664, args=0x60c72c8) at eval.c:2764
  #12 0x012246e8 in Fapply (nargs=2, args=0x829970) at eval.c:2343
  #13 0x01224f2d in apply1 (fn=-6917529027540600496, arg=-4611686018326943040)
      at eval.c:2559
  #14 0x0121f650 in Fmacroexpand (form=-4611686018326943056,
      environment=-4611686018327011424) at eval.c:1035
  #15 0x0122588f in Ffuncall (nargs=3, args=0x829b48) at eval.c:2718
  #16 0x0128959c in exec_byte_code (bytestr=-9223372036758992784,
      vector=-6917529027545329872, maxdepth=4611686018427387914,
      args_template=4611686018427388418, nargs=2, args=0x82a110)
      at bytecode.c:639
  #17 0x012268de in funcall_lambda (fun=-6917529027545333672, nargs=2,
      arg_vector=0x82a100) at eval.c:2879
  #18 0x01225e1f in Ffuncall (nargs=3, args=0x82a0f8) at eval.c:2764
  #19 0x0128959c in exec_byte_code (bytestr=-9223372036755022888,
      vector=-6917529027541352232, maxdepth=4611686018427387914,
      args_template=4611686018427388418, nargs=2, args=0x82a698)
      at bytecode.c:639
  #20 0x012268de in funcall_lambda (fun=-6917529027541352128, nargs=2,
      arg_vector=0x82a688) at eval.c:2879
  #21 0x01225e1f in Ffuncall (nargs=3, args=0x82a680) at eval.c:2764
  #22 0x0128959c in exec_byte_code (bytestr=-9223372036754938232,
      vector=-6917529027541242864, maxdepth=4611686018427387908,
      args_template=4611686018427388161, nargs=1, args=0x82abd8)
      at bytecode.c:639
  #23 0x012268de in funcall_lambda (fun=-6917529027541242840, nargs=1,
      arg_vector=0x82abd0) at eval.c:2879
  #24 0x01225e1f in Ffuncall (nargs=2, args=0x82abc8) at eval.c:2764
  #25 0x0128959c in exec_byte_code (bytestr=-9223372036754939440,
      vector=-6917529027541248760, maxdepth=4611686018427387909,
      args_template=4611686018427388161, nargs=1, args=0x82b180)
      at bytecode.c:639
  #26 0x012268de in funcall_lambda (fun=-6917529027541248584, nargs=1,
      arg_vector=0x82b178) at eval.c:2879
  #27 0x01225e1f in Ffuncall (nargs=2, args=0x82b170) at eval.c:2764
  #28 0x0128959c in exec_byte_code (bytestr=-9223372036754939488,
      vector=-6917529027541248536, maxdepth=4611686018427387920,
      args_template=4611686018427388161, nargs=1, args=0x82b848)
      at bytecode.c:639
  #29 0x012268de in funcall_lambda (fun=-6917529027541248088, nargs=1,
      arg_vector=0x82b840) at eval.c:2879
  #30 0x01225e1f in Ffuncall (nargs=2, args=0x82b838) at eval.c:2764
  #31 0x0128959c in exec_byte_code (bytestr=-9223372036754945616,
      vector=-6917529027541250008, maxdepth=4611686018427387920,
      args_template=4611686018427388417, nargs=1, args=0x82bf80)
      at bytecode.c:639
  #32 0x012268de in funcall_lambda (fun=-6917529027541249216, nargs=1,
      arg_vector=0x82bf78) at eval.c:2879
  #33 0x01225e1f in Ffuncall (nargs=2, args=0x82bf70) at eval.c:2764
  #34 0x0128959c in exec_byte_code (bytestr=-9223372036754832632,
      vector=-6917529027541134088, maxdepth=4611686018427387912,
      args_template=4611686018427388161, nargs=1, args=0x82c548)
      at bytecode.c:639
  #35 0x012268de in funcall_lambda (fun=-6917529027541133960, nargs=1,
      arg_vector=0x82c540) at eval.c:2879
  #36 0x01225e1f in Ffuncall (nargs=2, args=0x82c538) at eval.c:2764
  #37 0x0128959c in exec_byte_code (bytestr=-9223372036754832680,
      vector=-6917529027541140920, maxdepth=4611686018427387914,
      args_template=4611686018427388160, nargs=0, args=0x82cba8)
      at bytecode.c:639
  #38 0x012268de in funcall_lambda (fun=-6917529027541140760, nargs=0,
      arg_vector=0x82cba8) at eval.c:2879
  #39 0x01225e1f in Ffuncall (nargs=1, args=0x82cba0) at eval.c:2764
  #40 0x0128959c in exec_byte_code (bytestr=-9223372036757936560,
      vector=-6917529027544929360, maxdepth=4611686018427387927,
      args_template=4611686018427388161, nargs=1, args=0x82d5b8)
      at bytecode.c:639
  #41 0x012268de in funcall_lambda (fun=-6917529027544928504, nargs=1,
      arg_vector=0x82d5b0) at eval.c:2879
  #42 0x01225e1f in Ffuncall (nargs=2, args=0x82d5a8) at eval.c:2764
  #43 0x0128959c in exec_byte_code (bytestr=-9223372036760992488,
      vector=-6917529027547292792, maxdepth=4611686018427387925,
      args_template=4611686018427387904, nargs=0, args=0x82e258)
      at bytecode.c:639
  #44 0x012268de in funcall_lambda (fun=-6917529027547291064, nargs=0,
      arg_vector=0x82e258) at eval.c:2879
  #45 0x01225e1f in Ffuncall (nargs=1, args=0x82e250) at eval.c:2764
  #46 0x0128959c in exec_byte_code (bytestr=-9223372036756613024,
      vector=-6917529027542917320, maxdepth=4611686018427387916,
      args_template=4611686018427387904, nargs=0, args=0x82e850)
      at bytecode.c:639
  #47 0x012268de in funcall_lambda (fun=-6917529027542916696, nargs=0,
      arg_vector=0x82e850) at eval.c:2879
  #48 0x012263b2 in apply_lambda (fun=-6917529027542916696, args=0, count=21)
      at eval.c:2816
  #49 0x01223de3 in eval_sub (form=-4611686018332481616) at eval.c:2233
  #50 0x01222b56 in Feval (form=-4611686018332481616, lexical=0) at eval.c:2010
  #51 0x01223886 in eval_sub (form=-4611686018328424112) at eval.c:2191
  #52 0x0121c562 in Fprogn (body=-4611686018328424080) at eval.c:432
  #53 0x012232ef in eval_sub (form=-4611686018328424240) at eval.c:2141
  #54 0x01269693 in readevalloop (readcharfun=28056,
      stream=0x77c5fce0 <msvcrt!_iob+96>, sourcename=-9223372036838132000,
      printflag=false, unibyte=0, readfun=0, start=0, end=0) at lread.c:1980
  #55 0x01266f73 in Fload (file=-9223372036838132176, noerror=0, nomessage=0,
      nosuffix=0, must_suffix=0) at lread.c:1367
  #56 0x01223a45 in eval_sub (form=-4611686018340754848) at eval.c:2202
  #57 0x012205e2 in internal_lisp_condition_case (var=0,
      bodyform=-4611686018340754848, handlers=-4611686018340754768)
      at eval.c:1285
  #58 0x0121fe64 in Fcondition_case (args=-4611686018340754736) at eval.c:1211
  #59 0x012232ef in eval_sub (form=-4611686018340754720) at eval.c:2141
  #60 0x01222b56 in Feval (form=-4611686018340754720, lexical=0) at eval.c:2010
  #61 0x01155640 in top_level_2 () at keyboard.c:1127
  #62 0x01220675 in internal_condition_case (bfun=0x115560a <top_level_2>,
      handlers=21616, hfun=0x1154dc1 <cmd_error>) at eval.c:1314
  #63 0x011556a6 in top_level_1 (ignore=0) at keyboard.c:1135
  #64 0x0121f7fd in internal_catch (tag=57512, func=0x1155646 <top_level_1>,
      arg=0) at eval.c:1080
  #65 0x01155522 in command_loop () at keyboard.c:1096
  #66 0x011547f3 in recursive_edit_1 () at keyboard.c:703
  #67 0x01154a8f in Frecursive_edit () at keyboard.c:774
  #68 0x01152244 in main (argc=7, argv=0xa440d8) at emacs.c:1698

  Lisp Backtrace:
  "skkdic-extract-conversion-data" (0x829398)
  0x5fd3950 PVEC_COMPILED
  "macroexpand" (0x829b50)
  "macroexp-macroexpand" (0x82a100)
  "byte-compile-recurse-toplevel" (0x82a688)
  "byte-compile-toplevel-file-form" (0x82abd0)
  0x5f355b8 PVEC_COMPILED
  "byte-compile-from-buffer" (0x82b840)
  "byte-compile-file" (0x82bf78)
  "batch-byte-compile-file" (0x82c540)
  "batch-byte-compile" (0x82cba8)
  "command-line-1" (0x82d5b0)
  "command-line" (0x82e258)
  "normal-top-level" (0x82e850)
  "eval" (0x82eb30)
  "progn" (0x82ed20)
  "load" (0x82f500)
  "condition-case" (0x82f7d0)
  (gdb) fr 5
  #5  0x0122f3a1 in Fsubstring (string=-9223372036754289936,
      from=4611686018427390336, to=4611686018427390336) at fns.c:1282
  1282      validate_subarray (string, from, to, size, &ifrom, &ito);
  (gdb) pp string
  " s  "
  (gdb)

The error seems to come from this function in ja-dic-cnv.el:

  (defun skkdic-extract-conversion-data (entry)
    (string-match "^\\cj+[a-z]* " entry)
    (let ((kana (substring entry (match-beginning 0) (1- (match-end 0))))
	  (i (match-end 0))
	  candidates)
      (while (string-match "[^ ]+" entry i)
	(setq candidates (cons (match-string 0 entry) candidates))
	(setq i (match-end 0)))
      (cons (skkdic-get-kana-compact-codes kana) candidates)))

The call to 'substring' is the one that errors out.  So this again
points to some problem with categories, as "\\cj" is in the regexp.

> make[2]: Entering directory '/home/raeburn/dev/emacs/s/lisp'
> make -C ../leim all EMACS="../src/emacs"
> make[3]: Entering directory '/home/raeburn/dev/emacs/s/leim'
> /bin/mkdir -p ../lisp/leim/ja-dic
>   GEN      ../lisp/leim/ja-dic/ja-dic.el
> Loading ../src/dumped.elc...
> Reading file "/home/raeburn/dev/emacs/s/leim/SKK-DIC/SKK-JISYO.L" ...
> Processing OKURI-ARI entries ...
> Debugger entered--Lisp error: (search-failed "^\\cH")
>   re-search-forward("^\\cH")
>   (let ((from (point)) to) (search-forward ";; okuri-nasi") (beginning-of-line) (setq to (point)) (narrow-to-region from to) (skkdic-convert-okuri-ari skkbuf buf) (widen) (goto-char to) (forward-line 1) (setq from (point)) (re-search-forward "^\\cH") (setq to (match-beginning 0)) (narrow-to-region from to) (skkdic-convert-postfix skkbuf buf) (widen) (goto-char to) (skkdic-convert-prefix skkbuf buf) (skkdic-collect-okuri-nasi) (skkdic-convert-okuri-nasi skkbuf buf) (save-current-buffer (set-buffer buf) (goto-char (point-max)) (insert ";;\n(provide 'ja-dic)\n\n" ";; Local Variables:\n" ";; version-control: never\n" ";; no-update-autoloads: t\n" ";; coding: utf-8\n" ";; End:\n\n" ";;; ja-dic.el ends here\n")))

Not sure why I didn't see the error with okuri-nasi, perhaps the
previous build attempts already generated that.  If I do

  touch leim/SKK-DIC/SKK-JISYO.L

the next "make" indeed fails as on your system.

One other thing I noticed is that most of the *.elc files produced by
this build are different from those I see on master.  The differences
are sometimes just a few bytes (e.g., in mule-diag.elc), but sometimes
much larger (e.g., files.elc).  Perhaps this points to some subtle
problem in byte compilation?  But even if so, that cannot explain the
failure to compile eww.el and ja-dic.el.

HTH

Re: Skipping unexec via a big .elc file

[Andreas Schwab]

On Jan 14 2017, Eli Zaretskii <eliz@gnu.org> wrote:

> The line number in the error message is bogus, it points to a require
> line (that's a known issue with byte-compiler error reporting, I
> think).

It's not bogus, since the error was raised while the byte-compiler
evaluated the form there.  Lisp errors don't carry line number
information so there isn't much the byte-compiler can do.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Ken Raeburn <raeburn@raeburn.org>,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sat, 14 Jan 2017 11:55:42 +0100
> 
> On Jan 14 2017, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > The line number in the error message is bogus, it points to a require
> > line (that's a known issue with byte-compiler error reporting, I
> > think).
> 
> It's not bogus, since the error was raised while the byte-compiler
> evaluated the form there.

It's "bogus" in the sense that it isn't useful for finding the code
which triggered the error.

Re: Skipping unexec via a big .elc file

[Alan Mackenzie]

Hello, Eli.

On Sat, Jan 14, 2017 at 01:07:17PM +0200, Eli Zaretskii wrote:
> > From: Andreas Schwab <schwab@linux-m68k.org>
> > Cc: Ken Raeburn <raeburn@raeburn.org>,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> > Date: Sat, 14 Jan 2017 11:55:42 +0100

> > On Jan 14 2017, Eli Zaretskii <eliz@gnu.org> wrote:

> > > The line number in the error message is bogus, it points to a require
> > > line (that's a known issue with byte-compiler error reporting, I
> > > think).

> > It's not bogus, since the error was raised while the byte-compiler
> > evaluated the form there.

> It's "bogus" in the sense that it isn't useful for finding the code
> which triggered the error.

Just as a matter of interest, I spent quite a bit of time in the summer
trying to fix this.  My approach was this:
(i) The modified reader created an association list between each cons it
  creates and the source code position.
(ii) Each time a compiler function transformed such a cons, instead of
  the function returning the transformed form, it did setcar/setcdr into
  the original cons to preserve the mapping in the association table.
(iii) On emitting an error/warning, the compiler would look up the
  source code position in the association list.

I'm confident that such an approach would work.  However, it was an
enormous amount of work to adapt the compiler, and I got distracted by
other things, so haven't managed to produce anything workable, yet.  At
least there's already a reliable test suite for this, namely make
bootstrap.  :-)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Skipping unexec via a big .elc file

[Andreas Schwab]

On Jan 14 2017, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: Ken Raeburn <raeburn@raeburn.org>,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Sat, 14 Jan 2017 11:55:42 +0100
>> 
>> On Jan 14 2017, Eli Zaretskii <eliz@gnu.org> wrote:
>> 
>> > The line number in the error message is bogus, it points to a require
>> > line (that's a known issue with byte-compiler error reporting, I
>> > think).
>> 
>> It's not bogus, since the error was raised while the byte-compiler
>> evaluated the form there.
>
> It's "bogus" in the sense that it isn't useful for finding the code
> which triggered the error.

It is as bogus as every Lisp error.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Date: Sat, 14 Jan 2017 13:19:12 +0100
> Cc: raeburn@raeburn.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> >> > The line number in the error message is bogus, it points to a require
> >> > line (that's a known issue with byte-compiler error reporting, I
> >> > think).
> >> 
> >> It's not bogus, since the error was raised while the byte-compiler
> >> evaluated the form there.
> >
> > It's "bogus" in the sense that it isn't useful for finding the code
> > which triggered the error.
> 
> It is as bogus as every Lisp error.

On the contrary: most of them provide useful information about the
error locus.  This one clearly didn't.

Re: Skipping unexec via a big .elc file

[Andreas Schwab]

On Jan 14 2017, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Date: Sat, 14 Jan 2017 13:19:12 +0100
>> Cc: raeburn@raeburn.org, monnier@iro.umontreal.ca, emacs-devel@gnu.org
>> 
>> >> > The line number in the error message is bogus, it points to a require
>> >> > line (that's a known issue with byte-compiler error reporting, I
>> >> > think).
>> >> 
>> >> It's not bogus, since the error was raised while the byte-compiler
>> >> evaluated the form there.
>> >
>> > It's "bogus" in the sense that it isn't useful for finding the code
>> > which triggered the error.
>> 
>> It is as bogus as every Lisp error.
>
> On the contrary: most of them provide useful information about the
> error locus.  This one clearly didn't.

It accurately tells you the form that caused the error, something you
never get from a Lisp error.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: raeburn@raeburn.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sat, 14 Jan 2017 16:12:22 +0100
> 
> >> It is as bogus as every Lisp error.
> >
> > On the contrary: most of them provide useful information about the
> > error locus.  This one clearly didn't.
> 
> It accurately tells you the form that caused the error, something you
> never get from a Lisp error.

It's accurate, but utterly useless.

Re: Skipping unexec via a big .elc file

[Andreas Schwab]

On Jan 14 2017, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: raeburn@raeburn.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Sat, 14 Jan 2017 16:12:22 +0100
>> 
>> >> It is as bogus as every Lisp error.
>> >
>> > On the contrary: most of them provide useful information about the
>> > error locus.  This one clearly didn't.
>> 
>> It accurately tells you the form that caused the error, something you
>> never get from a Lisp error.
>
> It's accurate, but utterly useless.

No, it isn't.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

>   leim/ja-dic/ja-dic.el:76:1:Error: Args out of range: " s  ", 2432, 2432
[...]
> The line number in the error message is bogus, it points to a require
> line (that's a known issue with byte-compiler error reporting, I
> think).

I don't think it's "bogus": it says that the error occurred while
compiling that `require` line, i.e. while loading the corresponding file.
You can set byte-compile-debug (along with debug-on-error) to get
a backtrace which will be more useful.


        Stefan

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sat, 14 Jan 2017 10:30:45 -0500
> Cc: Ken Raeburn <raeburn@raeburn.org>, schwab@linux-m68k.org,
> 	emacs-devel@gnu.org
> 
> You can set byte-compile-debug (along with debug-on-error) to get
> a backtrace which will be more useful.

That doesn't help when one is presented with a build log.

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

>> You can set byte-compile-debug (along with debug-on-error) to get
>> a backtrace which will be more useful.
> That doesn't help when one is presented with a build log.

Not directly, no, indeed.  Usually I then fire an interactive Emacs, set
the vars and call byte-compile-file to reproduce the problem in an
environment where I can investigate the backtrace comfortably.

My point was simply that this is an *evaluation* error more than an error
in the compiled code, so the poverty of the info is due to the poverty
of info we get when running Elisp code (and this is indeed somewhat
linked to the byte-compiler since the byte-compiler doesn't preserve
the source location in the bytecode it emits).


        Stefan

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sat, 14 Jan 2017 13:11:26 -0500
> 
> >> You can set byte-compile-debug (along with debug-on-error) to get
> >> a backtrace which will be more useful.
> > That doesn't help when one is presented with a build log.
> 
> Not directly, no, indeed.  Usually I then fire an interactive Emacs, set
> the vars and call byte-compile-file to reproduce the problem in an
> environment where I can investigate the backtrace comfortably.

There's more than one way of tracking the real locus of the problem.
My point is that either way, it's an annoyance which makes
investigation of such problems significantly less efficient than when
the byte compiler points out the source file and the line number where
it happens, or close thereabouts (which is what happens most of the
time).

I gather that we are in violent agreement about that.

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

I think I may have figured out why I was getting crashes relating to the face cache but it wasn’t very reproducible.  Some of the face creation code paths will ensure that a cache exists for a frame before using it — like the handling of “menu” in internal-set-lisp-face-attribute — and some do not.  In a regular Emacs build, the order of operations in the C and Lisp code dictate the order in which face definitions are processed.  So, for example, in a batch-mode test invocation I tried, the “menu” face handling created the cache for frame “F1” before using it.

But using dumped.elc, face property settings get restored, but the code generated assumes that the order doesn’t matter, so the list of face names depends not just on which Lisp code was loaded, but on the order they’re seen under “mapatoms”, i.e., based on load order and the obarray size.  (So my Mac/NS and GNU/Linux/X11 builds have different lists of names, and different orders.)

I’m looking at internal-set-lisp-face-attribute as a place to always ensure the existence of the cache, but there may be a better location.

On Jan 14, 2017, at 05:41, Eli Zaretskii <eliz@gnu.org> wrote:
> [… much about failures I’m still looking at…]
> One other thing I noticed is that most of the *.elc files produced by
> this build are different from those I see on master.  The differences
> are sometimes just a few bytes (e.g., in mule-diag.elc), but sometimes
> much larger (e.g., files.elc).  Perhaps this points to some subtle
> problem in byte compilation?  But even if so, that cannot explain the
> failure to compile eww.el and ja-dic.el.

I built a couple versions, and found several .elc files different.  The first case I looked at was macroexp--const-symbol-p in macroexp.elc.  From disassembling, it appears that the expression “(boundp 'byte-compile-const-variables)” is optimized out in the build from the branch point, but not in the build including the dumped.elc changes.  I’m not sure why yet, but it’s almost certainly a bug that they’re different.  And a bug affecting the emacs-lisp environment and/or the byte compiler output could certainly cause later attempts at byte compilation (using newly byte-compiled code) to misbehave.

Ken

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

On Jan 21, 2017, at 02:58, Ken Raeburn <raeburn@raeburn.org> wrote:
> I built a couple versions, and found several .elc files different.  The first case I looked at was macroexp--const-symbol-p in macroexp.elc.  From disassembling, it appears that the expression “(boundp 'byte-compile-const-variables)” is optimized out in the build from the branch point, but not in the build including the dumped.elc changes.  I’m not sure why yet, but it’s almost certainly a bug that they’re different.  And a bug affecting the emacs-lisp environment and/or the byte compiler output could certainly cause later attempts at byte compilation (using newly byte-compiled code) to misbehave.

Ah, this may be a false alarm.  I’d overlooked the fact that the updated version (October 31) of Stefan’s patch changed that code to insert that expression on the branch, and I assumed the two were compiling the same source.  But if byte-compile-const-variables can be seen as unbound, that could also alter the optimization results compared to the master branch.  Perhaps that should be fixed, if possible.

Ken

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

On Jan 14, 2017, at 05:41, Eli Zaretskii <eliz@gnu.org> wrote:
> It does fail for me while byte compiling 2 files:

I still haven’t hit these.

> The category '>' is defined in characters.el.  Surprisingly,
> characters.elc in this branch is identical to the file on master, so
> byte compilation (see below) is off the hook here.  What else could
> explain that this category is deemed unknown?

I recently noticed the standard syntax and category tables don’t appear to be among the information dumped out, so anything set by characters.el isn’t preserved.  I also didn’t see an existing way to restore them trivially.  (A huge list of “modify-syntax-entry” calls and such seems impractical.)  Also, the buffer-local nature of some variables was being lost; that presented itself as a failure to get syntax-based highlighting in C source files.

Having patched around these, I’m still failing on the same file, but later; it prompts me for the coding system to use to write an output file, because it’s not valid UTF-8.  Apparently the leading comments copied from SKK-JISYO.L are being corrupted.  The first non-ASCII bytes in the buffer (in the “ACKNOWLEDGEMENTS” part of the comment) are 0xe3, 0x81, 0x93, 0xe3, 0x81, 0xae, 0xe8 in a normal build and 0xf5, 0x80, 0x84, 0xac, 0xf5, 0x80, 0x85, 0x87 in my build.  I discovered this maybe half an hour ago, so that’s as far as I’ve gotten.

I’ve just pushed my changes, plus your change to avoid the dump-emacs call, to the branch.  Saving and restoring the standard syntax table seems cheap, because it was already referenced by other objects that were dumped out, but the standard category table almost doubles the size of my dumped.elc, and presumably increases the time to read it accordingly.  Perhaps reading characters.el(c) at startup would be a better choice.

Ken

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Thu, 2 Feb 2017 04:10:38 -0500
> Cc: Andreas Schwab <schwab@linux-m68k.org>,
>  Stefan Monnier <monnier@iro.umontreal.ca>,
>  emacs-devel@gnu.org
> 
> Perhaps reading characters.el(c) at startup would be a better
> choice.

How about changing emacs.c to read characters.el(c) just after
dumped.elc?

Alternatively, would it be possible to simply append characters.elc to
the end of dumped.elc, as part of preparing the latter?

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

> On Feb 4, 2017, at 05:37, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Ken Raeburn <raeburn@raeburn.org>
>> Date: Thu, 2 Feb 2017 04:10:38 -0500
>> Cc: Andreas Schwab <schwab@linux-m68k.org>,
>> Stefan Monnier <monnier@iro.umontreal.ca>,
>> emacs-devel@gnu.org
>> 
>> Perhaps reading characters.el(c) at startup would be a better
>> choice.
> 
> How about changing emacs.c to read characters.el(c) just after
> dumped.elc?
> 
> Alternatively, would it be possible to simply append characters.elc to
> the end of dumped.elc, as part of preparing the latter?

For now, I changed loadup.el to emit a “load” form to get characters.elc at startup, and that seems to be working.  Copying the contents of characters.elc may be very slightly faster, but I haven’t done any timing tests.

I also tracked down my new ja-dic-cnv problem.  It looks like SKK-JISYO.L was being mangled on read because the input sequences weren’t recognized as Unicode compatible; this caused the resulting buffer not to be considered UTF-8 compatible, so it prompted for a coding system to write with.  Calling unify-charset on the various charsets seems to be needed.

With that change, I’m able to run “make bootstrap” in a GNU/Linux/X11 configuration and it runs to completion.  I haven’t yet tested it on macOS, or compared the .elc for the differences you were describing.

Ken

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Sun, 5 Feb 2017 09:19:38 -0500
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> For now, I changed loadup.el to emit a “load” form to get characters.elc at startup, and that seems to be working.  Copying the contents of characters.elc may be very slightly faster, but I haven’t done any timing tests.
> 
> I also tracked down my new ja-dic-cnv problem.  It looks like SKK-JISYO.L was being mangled on read because the input sequences weren’t recognized as Unicode compatible; this caused the resulting buffer not to be considered UTF-8 compatible, so it prompted for a coding system to write with.  Calling unify-charset on the various charsets seems to be needed.
> 
> With that change, I’m able to run “make bootstrap” in a GNU/Linux/X11 configuration and it runs to completion.  I haven’t yet tested it on macOS, or compared the .elc for the differences you were describing.

Thanks.  Are those changes committed to the branch?

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

On Feb 5, 2017, at 10:51, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> Thanks.  Are those changes committed to the branch?

Yes, I pushed them this morning.

Ken

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

And now I’ve got a *possible* explanation for why I’m seeing differences in some .elc files.

It appears that the .elc output can vary depending on whether other loaded Lisp code was compiled or not.

I found differences in diary-lib.elc between two of my build trees (raeburn-startup branch, and the branch point).  In the first function I looked at, the differences came down to this:

335	constant  (12 31 -1)   constant  (12 31 -1)
336	dup	  	       dup	  	    
337	varbind	  date	       varbind	  date	    
339	dup	  	       varbind	  date	    
340	varbind	  date	       constant  (12 31 -1)
342	car	  	       car	  	    
343	unbind	  1	       unbind	  1

The constant here comes from calendar-absolute-from-gregorian, a defsubst in calendar.el.  I tried with one source base (at the branch point), compiling diary-lib.el with calendar.elc present, and again with calendar.elc missing so that calendar.el would get used.  The generated .elc files showed the same differences.

This is arguably a bug, but not one added by the big-elc changes.

I almost always build with a make option like “-j4”, so the timing of byte compilations of different files relative to one another isn’t entirely predictable.  This *could* account for a lot of differences I’m seeing.  It’s not a sure thing that this sort of thing is the only cause of differences; I’ll have to do a fully serialized bootstrap of both versions of the code to see.

Ken

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

> It appears that the .elc output can vary depending on whether other loaded
> Lisp code was compiled or not.

Indeed: the culprit is the defsubst implementation.  Currently, if
a function is byte-compiled, the optimizer inlines its byte-codes
and when it's not yet byte-compiled, then it inlines the source code.

We should probably change that so that when it finds that the defsubst
function is not yet byte-compiled, it byte-compiles it and then inlines
the resulting byte-codes.


        Stefan

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

> On Feb 6, 2017, at 10:39, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>> It appears that the .elc output can vary depending on whether other loaded
>> Lisp code was compiled or not.
> 
> Indeed: the culprit is the defsubst implementation.  Currently, if
> a function is byte-compiled, the optimizer inlines its byte-codes
> and when it's not yet byte-compiled, then it inlines the source code.
> 
> We should probably change that so that when it finds that the defsubst
> function is not yet byte-compiled, it byte-compiles it and then inlines
> the resulting byte-codes.

Is this a known (and filed) bug?  A quick search for defsubst in debbugs only finds me one unrelated report.

In any case, doing “make bootstrap” from clean trees (which I’m assuming will byte-compile files in the same order each time) still gets me a few differences between the branch point and the branch, including python.elc differing in use of dynamic docstrings, and url-handler.elc file-name-handler wrappers saying “no original documentation”.  Still more to debug, I guess.

Ken

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

> Is this a known (and filed) bug?

I don't think it's filed, no.  I've known about it for a while now, and
it came up "recently" in the discussion about reproducible builds.
Until then it wasn't considered as a real bug, I think, more like
a quirk.

> In any case, doing “make bootstrap” from clean trees (which I’m assuming
> will byte-compile files in the same order each time)

Not sure if make guarantees a specific order of execution in that case,
but in my experience I think it does operate in a deterministic way, indeed.


        Stefan

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

On Feb 6, 2017, at 17:39, Stefan Monnier <monnier@IRO.UMontreal.CA> wrote:

>> Is this a known (and filed) bug?
> 
> I don't think it's filed, no.  I've known about it for a while now, and
> it came up "recently" in the discussion about reproducible builds.
> Until then it wasn't considered as a real bug, I think, more like
> a quirk.

Ah, okay.  I didn’t follow that discussion closely.  I haven’t got the bandwidth to keep up on everything, and until now I thought I didn’t care about this one. :-)

With my bootstrap builds running without parallel make, I’ve gotten things much further along in terms of generating .elc files that match what I get without all the big-elc changes.

The difference in progmodes/python.elc came down to the use of UTF-8 in the environment during byte compilation affecting the generated doc strings (using format-message in a macro).  Removing internal--text-quoting-flag from the stuff saved in dumped.elc made the files match for me on my Mac (with UTF-8 in use by default).  I think that’s just papering over the real problem (the macro’s result shouldn’t depend on the UTF-8-ness of the environment), but the flag should reflect the environment of the current Emacs invocation anyway, not the one that produced dumped.elc.

The difference in url/url-handler.elc was because the subr doc strings were getting lost.  The numbers (“DOC” file offsets) stored in the Lisp_Subr structure weren’t preserved, so url-handlers-create-wrapper would just fill in “No original documentation.”  I’m making dumped.elc invoke Snarf-documentation for now.

A tangent: As it happens, a couple years back I was experimenting with having C-based subr/variable documentation stuffed into the executable instead of needing the DOC file, in ways that wouldn’t add a lot of Lisp data unless the doc strings were actually needed.  For subr documentation, it doesn’t create Lisp strings until they’re requested.  For variables, I’ve got an idea on deferring the Lisp string creation, but currently they’re created at startup and stuffed into the property list.  I’ve just updated it to recent Emacs sources, in case we might want to explore that direction further; it might be more efficient than patching up doc pointers every time we start up.

Anyway, with the changes I’ve just pushed to the branch, my bootstrapped tree has .elc files that match those built from the branch point, except for mule.elc, macroexp.elc (both source files changed on the branch), bytecomp.elc and byte-opt.elc (probably due to macroexp changes).

I haven’t tried any more extensive testing.
There may be some funny stuff going on in restoring the charset definitions that I still need to look into.

I haven’t pulled in Ken Brown’s Cygwin changes; Ken, feel free to push those to the branch as well.

Ken

Re: Skipping unexec via a big .elc file

[Ken Brown]

On 2/8/2017 5:31 AM, Ken Raeburn wrote:
> I haven’t pulled in Ken Brown’s Cygwin changes; Ken, feel free to push those to the branch as well.

Done.

Ken

Re: Skipping unexec via a big .elc file

[Ken Brown]

[-- Attachment #1: Type: text/plain, Size: 230 bytes --]

On 2/5/2017 9:19 AM, Ken Raeburn wrote:
> With that change, I’m able to run “make bootstrap” in a GNU/Linux/X11 configuration and it runs to completion.

The attached patch enables the build to succeed on Cygwin.

Ken

[-- Attachment #2: 0001-Fix-build-on-Cygwin.patch --]
[-- Type: text/plain, Size: 1518 bytes --]

From 165c3356ebb5413277197e4e17c97e7758f96396 Mon Sep 17 00:00:00 2001
From: Ken Brown <kbrown@cornell.edu>
Date: Sun, 5 Feb 2017 14:58:59 -0500
Subject: [PATCH] Fix build on Cygwin

* configure.ac: Use system malloc on Cygwin.

* lisp/loadup.el: Use ".exe" suffix on Cygwin.
---
 configure.ac   | 4 +---
 lisp/loadup.el | 2 +-
 2 files changed, 2 insertions(+), 4 deletions(-)

diff --git a/configure.ac b/configure.ac
index 425e338..c1fd14d 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2158,9 +2158,7 @@ AC_DEFUN
 test "$CANNOT_DUMP" = yes ||
 case "$opsys" in
   ## darwin ld insists on the use of malloc routines in the System framework.
-  darwin | mingw32 | nacl | sol2-10) ;;
-  cygwin) hybrid_malloc=yes
-          system_malloc= ;;
+  cygwin | darwin | mingw32 | nacl | sol2-10) ;;
   *) test "$ac_cv_func_sbrk" = yes && system_malloc=$emacs_cv_sanitize_address;;
 esac
 
diff --git a/lisp/loadup.el b/lisp/loadup.el
index 80e9a28..72f24a6 100644
--- a/lisp/loadup.el
+++ b/lisp/loadup.el
@@ -455,7 +455,7 @@
       ;; other GNU program's build process.
       ;; (dump-emacs "emacs" "temacs")
       ;; (message "%d pure bytes used" pure-bytes-used)
-      (let ((exe (if (memq system-type '(windows-nt ms-dos)) ".exe" "")))
+      (let ((exe (if (memq system-type '(cygwin windows-nt ms-dos)) ".exe" "")))
         (copy-file (expand-file-name (concat "temacs" exe) invocation-directory)
                    (expand-file-name (concat "emacs" exe) invocation-directory)
                    t)
-- 
2.8.3

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Sun, 5 Feb 2017 09:19:38 -0500
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> I also tracked down my new ja-dic-cnv problem.  It looks like SKK-JISYO.L was being mangled on read because the input sequences weren’t recognized as Unicode compatible; this caused the resulting buffer not to be considered UTF-8 compatible, so it prompted for a coding system to write with.  Calling unify-charset on the various charsets seems to be needed.

Is this part in the repository?  Because I still get prompted for an
encoding when producing ja-dic.el:

    GEN      ../lisp/leim/ja-dic/ja-dic.el
  Reading file "d:/gnu/git/emacs/no-unexec/leim/SKK-DIC/SKK-JISYO.L" ...
  Processing OKURI-ARI entries ...
  Processing POSTFIX entries ...
  Processing PREFIX entries ...
  Collecting OKURI-NASI entries ...
  collected 26% ...
  collected 30% ...
  collected 40% ...
  collected 50% ...
  collected 60% ...
  collected 70% ...
  collected 80% ...
  collected 90% ...
  Processing OKURI-NASI entries ...
  processed 10% ...
  processed 20% ...
  processed 30% ...
  processed 40% ...
  processed 50% ...
  processed 60% ...
  processed 70% ...
  processed 80% ...
  processed 90% ...
  processed 100% ...
  Select coding system (default japanese-shift-jis): utf-8-unix

I needed to type utf-8-unix by hand.  Any ideas?  Is it possible that
this happens because my default encoding is not UTF-8?

I also pushed a small Windows-specific change to the branch, to allow
Windows users try building this branch.

Thanks.

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> Date: Sat, 25 Feb 2017 16:52:12 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> Is this part in the repository?  Because I still get prompted for an
> encoding when producing ja-dic.el:
> 
>     GEN      ../lisp/leim/ja-dic/ja-dic.el

Also, it looks like the logic in startup.el that should bypass certain
stuff under -Q isn't working, because I see my abbrevs being loaded
even though I invoked "emacs -Q".  Thoughts?

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

> On Feb 25, 2017, at 09:52, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Ken Raeburn <raeburn@raeburn.org>
>> Date: Sun, 5 Feb 2017 09:19:38 -0500
>> Cc: Emacs developers <emacs-devel@gnu.org>
>> 
>> I also tracked down my new ja-dic-cnv problem.  It looks like SKK-JISYO.L was being mangled on read because the input sequences weren’t recognized as Unicode compatible; this caused the resulting buffer not to be considered UTF-8 compatible, so it prompted for a coding system to write with.  Calling unify-charset on the various charsets seems to be needed.
> 
> Is this part in the repository?  Because I still get prompted for an
> encoding when producing ja-dic.el:

Yes, change d864464 has the unify-charset changes.



> 
>    GEN      ../lisp/leim/ja-dic/ja-dic.el
>  Reading file "d:/gnu/git/emacs/no-unexec/leim/SKK-DIC/SKK-JISYO.L" ...
>  Processing OKURI-ARI entries ...
>  Processing POSTFIX entries ...
>  Processing PREFIX entries ...
>  Collecting OKURI-NASI entries ...
>  collected 26% ...
>  collected 30% ...
>  collected 40% ...
>  collected 50% ...
>  collected 60% ...
>  collected 70% ...
>  collected 80% ...
>  collected 90% ...
>  Processing OKURI-NASI entries ...
>  processed 10% ...
>  processed 20% ...
>  processed 30% ...
>  processed 40% ...
>  processed 50% ...
>  processed 60% ...
>  processed 70% ...
>  processed 80% ...
>  processed 90% ...
>  processed 100% ...
>  Select coding system (default japanese-shift-jis): utf-8-unix
> 
> I needed to type utf-8-unix by hand.  Any ideas?  Is it possible that
> this happens because my default encoding is not UTF-8?

Looks like my environment has LANG=en_US.UTF-8, on Mac and GNU/Linux.  But setting LANG=C or en_US.ISO8859-1 doesn’t seem to cause the build to get hung up this way for me.

Did you do a full bootstrap after updating?  An outdated dumped.elc could certainly do this, and I know at least some of the dependencies aren’t current with the changes on the branch.  (I’ve taken to going as far as “git clean -f -d -x”, then using autogen.sh, configure, and “make bootstrap”, fairly often.)

> I also pushed a small Windows-specific change to the branch, to allow
> Windows users try building this branch.

Great!

> Also, it looks like the logic in startup.el that should bypass certain
> stuff under -Q isn't working, because I see my abbrevs being loaded
> even though I invoked "emacs -Q".  Thoughts?

Strange… this is also working for me.  At least, settings from my .emacs aren’t being applied, when I use “emacs -Q”.

Ken

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Sun, 26 Feb 2017 07:37:56 -0500
> Cc: emacs-devel@gnu.org
> 
> >    GEN      ../lisp/leim/ja-dic/ja-dic.el
> >  Reading file "d:/gnu/git/emacs/no-unexec/leim/SKK-DIC/SKK-JISYO.L" ...
> >  Processing OKURI-ARI entries ...
> >  Processing POSTFIX entries ...
> >  Processing PREFIX entries ...
> >  Collecting OKURI-NASI entries ...
> >  collected 26% ...
> >  collected 30% ...
> >  collected 40% ...
> >  collected 50% ...
> >  collected 60% ...
> >  collected 70% ...
> >  collected 80% ...
> >  collected 90% ...
> >  Processing OKURI-NASI entries ...
> >  processed 10% ...
> >  processed 20% ...
> >  processed 30% ...
> >  processed 40% ...
> >  processed 50% ...
> >  processed 60% ...
> >  processed 70% ...
> >  processed 80% ...
> >  processed 90% ...
> >  processed 100% ...
> >  Select coding system (default japanese-shift-jis): utf-8-unix
> > 
> > I needed to type utf-8-unix by hand.  Any ideas?  Is it possible that
> > this happens because my default encoding is not UTF-8?
> 
> Looks like my environment has LANG=en_US.UTF-8, on Mac and GNU/Linux.  But setting LANG=C or en_US.ISO8859-1 doesn’t seem to cause the build to get hung up this way for me.
> 
> Did you do a full bootstrap after updating?  An outdated dumped.elc could certainly do this, and I know at least some of the dependencies aren’t current with the changes on the branch.  (I’ve taken to going as far as “git clean -f -d -x”, then using autogen.sh, configure, and “make bootstrap”, fairly often.)

I've bootstrapped now, and this problem is gone.  Thanks.

> > Also, it looks like the logic in startup.el that should bypass certain
> > stuff under -Q isn't working, because I see my abbrevs being loaded
> > even though I invoked "emacs -Q".  Thoughts?
> 
> Strange… this is also working for me.  At least, settings from my .emacs aren’t being applied, when I use “emacs -Q”.

This problem is still there.  It has nothing to do with loading
~/.emacs, though: startup.el always loads your ~/.emacs.d/abbrev_defs,
if that file exists.  I'm not sure why it loads that file, but I
verified that the master version does that as well.

So the issue here is not that the file is loaded, but how it is
processed.  I only noticed this because my abbrev_defs file uses a
function that is only defined in my .emacs.  So "emacs -Q" on the
raeburn-startup branch barfs because that function is not known.
Strangely, "emacs -Q" on the master branch doesn't signal an error,
and I don't even see Fsignal called if I set a breakpoint there.  I
don't (yet) understand why the different behavior.

If you insert into your abbrev_defs file something that references a
function which is not defined, do you see the same problem as I do?

Btw, one thing that I saw while debugging is that purify-flag is set
to t while running the startup code.  This is because init_alloc_once
is called during startup (previously, it was only called by temacs).
I don't know if this is related to the issue (setting purify-flag to
nil in Frecursive_edit didn't help), but I thought I'd bring it up,
because maybe we need to set it to nil earlier.

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

On Mar 4, 2017, at 09:23, Eli Zaretskii <eliz@gnu.org> wrote:

>>> Also, it looks like the logic in startup.el that should bypass certain
>>> stuff under -Q isn't working, because I see my abbrevs being loaded
>>> even though I invoked "emacs -Q".  Thoughts?
>> 
>> Strange… this is also working for me.  At least, settings from my .emacs aren’t being applied, when I use “emacs -Q”.
> 
> This problem is still there.  It has nothing to do with loading
> ~/.emacs, though: startup.el always loads your ~/.emacs.d/abbrev_defs,
> if that file exists.  I'm not sure why it loads that file, but I
> verified that the master version does that as well.

Odd, seems like -Q should skip that, with the rest of the user’s initializations.

> 
> So the issue here is not that the file is loaded, but how it is
> processed.  I only noticed this because my abbrev_defs file uses a
> function that is only defined in my .emacs.  So "emacs -Q" on the
> raeburn-startup branch barfs because that function is not known.
> Strangely, "emacs -Q" on the master branch doesn't signal an error,
> and I don't even see Fsignal called if I set a breakpoint there.  I
> don't (yet) understand why the different behavior.
> 
> If you insert into your abbrev_defs file something that references a
> function which is not defined, do you see the same problem as I do?

I added a line:

  (missing-function)

in between some define-abbrev-table invocations, and “emacs -Q” on master (2-3 weeks old) and raeburn-startup both complain about it for me.

> Btw, one thing that I saw while debugging is that purify-flag is set
> to t while running the startup code.  This is because init_alloc_once
> is called during startup (previously, it was only called by temacs).
> I don't know if this is related to the issue (setting purify-flag to
> nil in Frecursive_edit didn't help), but I thought I'd bring it up,
> because maybe we need to set it to nil earlier.

I’ve been thinking that the branch should probably set CANNOT_DUMP unconditionally.  The behavior around pure storage and such under CANNOT_DUMP is probably closer to what we want for the branch.  But there’s at least one bug in building with CANNOT_DUMP for macOS I’ve got to clear up first.  As you say, it may not be at all related to the problem you’re running into, though.

Ken

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Mon, 6 Mar 2017 03:46:17 -0500
> Cc: emacs-devel@gnu.org
> 
> On Mar 4, 2017, at 09:23, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> >>> Also, it looks like the logic in startup.el that should bypass certain
> >>> stuff under -Q isn't working, because I see my abbrevs being loaded
> >>> even though I invoked "emacs -Q".  Thoughts?
> >> 
> >> Strange… this is also working for me.  At least, settings from my .emacs aren’t being applied, when I use “emacs -Q”.
> > 
> > This problem is still there.  It has nothing to do with loading
> > ~/.emacs, though: startup.el always loads your ~/.emacs.d/abbrev_defs,
> > if that file exists.  I'm not sure why it loads that file, but I
> > verified that the master version does that as well.
> 
> Odd, seems like -Q should skip that, with the rest of the user’s initializations.

Maybe so, but this code has been there since about forever, and the
documentation of -Q doesn't say user's abbrevs are bypassed, only
under -batch.  In any case, it's a separate problem.

> > So the issue here is not that the file is loaded, but how it is
> > processed.  I only noticed this because my abbrev_defs file uses a
> > function that is only defined in my .emacs.  So "emacs -Q" on the
> > raeburn-startup branch barfs because that function is not known.
> > Strangely, "emacs -Q" on the master branch doesn't signal an error,
> > and I don't even see Fsignal called if I set a breakpoint there.  I
> > don't (yet) understand why the different behavior.
> > 
> > If you insert into your abbrev_defs file something that references a
> > function which is not defined, do you see the same problem as I do?
> 
> I added a line:
> 
>   (missing-function)
> 
> in between some define-abbrev-table invocations, and “emacs -Q” on master (2-3 weeks old) and raeburn-startup both complain about it for me.

I debugged this some more: this has nothing to do with unknown
functions, you just need to have global abbrevs in the abbrev_defs
file, for example:

  (define-abbrev-table 'global-abbrev-table
    '(
      ("abbout" "about" nil 0)
      ("abotu" "about" nil 0)))

The problem seems to be that global-abbrev-table is not an abbrev
table where startup.el calls quietly-read-abbrev-file (abbrev-table-p
returns nil for it).  If I make this change:

diff --git a/lisp/startup.el b/lisp/startup.el
index 4a04f9c..7f55962 100644
--- a/lisp/startup.el
+++ b/lisp/startup.el
@@ -1263,6 +1263,8 @@ command-line
 	      (deactivate-mark)))
 
 	;; If the user has a file of abbrevs, read it (unless -batch).
+	(or (abbrev-table-p global-abbrev-table)
+	    (setq global-abbrev-table (make-abbrev-table)))
 	(when (and (not noninteractive)
 		   (file-exists-p abbrev-file-name)
 		   (file-readable-p abbrev-file-name))

then "emacs -Q" starts up normally.  Can you reproduce this?

global-abbrev-table is defined in abbrev.el like this:

  (defvar global-abbrev-table (make-abbrev-table)
    "The abbrev table whose abbrevs affect all buffers.
  Each buffer may also have a local abbrev table.
  If it does, the local table overrides the global one
  for any particular abbrev defined in both.")

So I think the issue could be that this defvar somehow doesn't end up
in dumped.elc as an abbrev table under the new build procedure.  Does that make sense?

Re: Skipping unexec via a big .elc file

[Andreas Schwab]

On Mär 11 2017, Eli Zaretskii <eliz@gnu.org> wrote:

> So I think the issue could be that this defvar somehow doesn't end up
> in dumped.elc as an abbrev table under the new build procedure.  Does that make sense?

I think the problem is that an abbrev table is actually an obarray,
which does not have a suitable print syntax.

ELISP> global-abbrev-table
[## 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]

ELISP> (abbrev-table-p [## 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0])
nil

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Ken Raeburn <raeburn@raeburn.org>,  emacs-devel@gnu.org
> Date: Sat, 11 Mar 2017 14:18:28 +0100
> 
> On Mär 11 2017, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > So I think the issue could be that this defvar somehow doesn't end up
> > in dumped.elc as an abbrev table under the new build procedure.  Does that make sense?
> 
> I think the problem is that an abbrev table is actually an obarray,
> which does not have a suitable print syntax.

Indeed, makes perfect sense.  Thanks.

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

> I think the problem is that an abbrev table is actually an obarray,
> which does not have a suitable print syntax.

Maybe now would be a good time to change the representation of
abbrev-tables?


        Stefan

Re: Skipping unexec via a big .elc file

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Maybe now would be a good time to change the representation of
  > abbrev-tables?

How would we prefer for them to print?

I am not convinced that it would be useful or convenient
to have them print out in a way that describes their contents.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

>> Maybe now would be a good time to change the representation of
>> abbrev-tables?
> How would we prefer for them to print?

I'm not talking about "representation" in the sense of "print format"
but in terms of which data-structure to use for them.  But yes, of
course that will affect the way they print.  Clearly, I'd hope that the
new representation would print `read'ably.

> I am not convinced that it would be useful or convenient
> to have them print out in a way that describes their contents.

The way abbrev-tables print right now is just plain bad: it's neither
computer-readable nor human-readable.


        Stefan

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

On Mar 11, 2017, at 08:18, Andreas Schwab <schwab@linux-m68k.org> wrote:

> On Mär 11 2017, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> So I think the issue could be that this defvar somehow doesn't end up
>> in dumped.elc as an abbrev table under the new build procedure.  Does that make sense?
> 
> I think the problem is that an abbrev table is actually an obarray,
> which does not have a suitable print syntax.

Ah, yes.  Thanks for noticing that.

And just yesterday I was thinking how convenient — and surprising — it was that we didn’t have to dump out any obarray objects; oh well.  Unless we’re going to arrange for obarrays to be printable and readable in a useful form, they’ll need special-casing.  But abbrev variables should be easy enough to recognize and process.  I’ll take a look.

Ken

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

> And just yesterday I was thinking how convenient — and surprising — it was
> that we didn’t have to dump out any obarray objects; oh well.  Unless we’re
> going to arrange for obarrays to be printable and readable in a useful form,
> they’ll need special-casing.  But abbrev variables should be easy enough to
> recognize and process.  I’ll take a look.

My personal favorite choice is to deprecate obarrays (most uses would
be better served by a hash-table), but getting rid of them completely is
rather tricky.  So we probably want to solve the obarray problem
regardless of whether we deprecate them.

It seems fairly, easy, tho:
- add a `make-obarray` function, which basically does the same as
  `make-vector` but uses another tag.  Use it in abbrev.el (and other
  applicable places).
- change `intern` and friends to accept those other kinds of vectors.
- change print.c to do something more clever with obarrays.
- deprecate use of plain vectors as obarrays.

I'm in the mood for procrastinating, so don't be surprised if a patch
shows up,


        Stefan

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

I have a patch which seems to recreate all the abbrev tables that were in the initial Emacs process, including the sharing (lisp-mode-abbrev-table being a parent of emacs-lisp-mode-abbrev-table; local-abbrev-table set to the fundamental mode table).  Please let me know if it fixes your problem.

It depends on a couple key things: (1) The abbrev tables are empty at that point, so we don’t have to worry about reconstructing all the abbrevs in a table; this can be fixed, it’s just tedious. (2) Abbrev-table values are only used as symbol values, or parents of other abbrev tables.  This is much harder.  Stefan’s printable replacement for obarrays would probably be a better solution.  Though, normally I’d expect people to want printing of an obarray to show symbol names, and for this use case we need the function, value, and plist data as well.

Ken


[-- Attachment #2: abbrev-table-patch --]
[-- Type: application/octet-stream, Size: 4844 bytes --]

commit dd4d7941a3915d16a0f78ab547e3656ec95f4f29
Author: Ken Raeburn <raeburn@raeburn.org>
Date:   Mon Mar 13 03:21:53 2017 -0400

    Dump and restore empty abbrev tables.
    
    Abbrev tables are obarrays and thus don't print out in a useful form.
    They need to be assembled at load time.  Fortunately, loadup.el only
    gives us empty abbrev tables, so we don't have to actually restore any
    abbrevs, only the tables.
    
    * lisp/loadup.el: When variable values are abbrev tables, emit a
    "make-abbrev-table" initialization with the appropriate property
    lists.  Check abbrev tables and their parents for instances of
    sharing.  Reject any abbrev tables that are not empty.

diff --git a/lisp/loadup.el b/lisp/loadup.el
index cc9ed7be1a..48a1208ed7 100644
--- a/lisp/loadup.el
+++ b/lisp/loadup.el
@@ -484,6 +484,10 @@
             (coding-systems '()) (coding-system-aliases '())
             (charsets '()) (charset-aliases '())
             (unified-charsets '())
+            (abbrev-tables (make-hash-table :test 'eq))
+            (abbrev-assign-cmds '())
+            (abbrev-make-cmds '())
+            (abbrev-counter 0)
             (cmds '()))
         (setcdr global-buffers-menu-map nil) ;; Get rid of buffer objects!
         (push `(internal--set-standard-syntax-table
@@ -539,6 +543,42 @@
                           '(let ((ol (make-overlay (point-min) (point-min))))
                              (delete-overlay ol)
                              ol))
+                         ;; abbrev-table-p isn't very defensive
+                         ((condition-case nil
+                              (abbrev-table-p v)
+                            (error nil))
+                          (cl-labels ((replace-abbrevs-for-dump
+                                       (table)
+                                       (or (abbrev-table-empty-p table)
+                                           (error "Non-empty abbrev tables not handled"))
+                                       (let ((newval (gethash table abbrev-tables)))
+                                         (if newval
+                                             `(aref scratch-abbrev-tables ,newval)
+                                           (let* ((props (symbol-plist (obarray-get table ""))))
+                                             (cond ((plist-get props :parents)
+                                                    (setq props (copy-sequence props))
+                                                    (plist-put props
+                                                               :parents
+                                                               (mapcar (lambda (value)
+                                                                         (replace-abbrevs-for-dump value))
+                                                                       (plist-get props :parents))))
+                                                   ((eq (length props) 2)
+                                                    ;; Only :abbrev-table-modiff, which gets added at creation anyway.
+                                                    (setq props nil)))
+                                             (push `(aset scratch-abbrev-tables
+                                                          ,abbrev-counter
+                                                          (make-abbrev-table ',props))
+                                                   abbrev-make-cmds)
+                                             (puthash table abbrev-counter abbrev-tables)
+                                             (prog1
+                                                 `(aref scratch-abbrev-tables ,abbrev-counter)
+                                               (setq abbrev-counter (1+ abbrev-counter))))))))
+                            (push `(set-default ',s
+                                                ,(replace-abbrevs-for-dump v))
+                                  abbrev-assign-cmds))
+                          ;; Placeholder to be used before we know
+                          ;; we've defined make-abbrev-table.
+                          0)
                          (v (macroexp-quote v))))
                      cmds)
                ;; Local variables: make-variable-buffer-local,
@@ -591,6 +631,10 @@
             (print '(get-buffer-create "*Messages*"))
             (print `(progn . ,cmds))
             (terpri)
+            ;; Now that make-abbrev-table is defined, use it.
+            (print `(let ((scratch-abbrev-tables (make-vector ,abbrev-counter 0)))
+                      ,@(nreverse abbrev-make-cmds)
+                      ,@abbrev-assign-cmds))
             (print `(let ((css ',charsets))
                       (dotimes (i 3)
                         (dolist (cs (prog1 css (setq css nil)))

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Mon, 13 Mar 2017 04:25:19 -0400
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> I have a patch which seems to recreate all the abbrev tables that were in the initial Emacs process, including the sharing (lisp-mode-abbrev-table being a parent of emacs-lisp-mode-abbrev-table; local-abbrev-table set to the fundamental mode table).  Please let me know if it fixes your problem.

Sorry for the long delay: Life™ intervened big time...

> It depends on a couple key things: (1) The abbrev tables are empty at that point, so we don’t have to worry about reconstructing all the abbrevs in a table; this can be fixed, it’s just tedious. (2) Abbrev-table values are only used as symbol values, or parents of other abbrev tables.  This is much harder.  Stefan’s printable replacement for obarrays would probably be a better solution.  Though, normally I’d expect people to want printing of an obarray to show symbol names, and for this use case we need the function, value, and plist data as well.

I applied your patch, and while dumping I get an error message:

  Dumping into dumped.elc...preparing...
  Dumping into dumped.elc...generating...
  Symbol's function definition is void: cl-labels

and dumped.elc is not re-created.  What did I miss?

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

On Mar 26, 2017, at 12:44, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Ken Raeburn <raeburn@raeburn.org>
>> Date: Mon, 13 Mar 2017 04:25:19 -0400
>> Cc: Emacs developers <emacs-devel@gnu.org>
>> 
>> I have a patch which seems to recreate all the abbrev tables that were in the initial Emacs process, including the sharing (lisp-mode-abbrev-table being a parent of emacs-lisp-mode-abbrev-table; local-abbrev-table set to the fundamental mode table).  Please let me know if it fixes your problem.
> 
> Sorry for the long delay: Life™ intervened big time…

It happens.  No worries.

> 
>> It depends on a couple key things: (1) The abbrev tables are empty at that point, so we don’t have to worry about reconstructing all the abbrevs in a table; this can be fixed, it’s just tedious. (2) Abbrev-table values are only used as symbol values, or parents of other abbrev tables.  This is much harder.  Stefan’s printable replacement for obarrays would probably be a better solution.  Though, normally I’d expect people to want printing of an obarray to show symbol names, and for this use case we need the function, value, and plist data as well.
> 
> I applied your patch, and while dumping I get an error message:
> 
>  Dumping into dumped.elc...preparing...
>  Dumping into dumped.elc...generating...
>  Symbol's function definition is void: cl-labels
> 
> and dumped.elc is not re-created.  What did I miss?

Looks like I missed a “require” or “load” to pull in cl-macs.  Perhaps it’s loaded by something else in my build that’s platform-dependent (X11 vs Windows?) and isn’t in yours; I’m not sure. But it isn’t working for me to just load it explicitly without fixing up the load path too.  Perhaps I should’ve just defined a helper function instead of using cl-labels.

For now, try adding this patch. It bootstraps for me, and should get cl-labels defined.

diff --git a/lisp/loadup.el b/lisp/loadup.el
index 4ef9712ab6..f9251020cd 100644
--- a/lisp/loadup.el
+++ b/lisp/loadup.el
@@ -57,6 +57,17 @@
 ;; Add subdirectories to the load-path for files that might get
 ;; autoloaded when bootstrapping.
 ;; This is because PATH_DUMPLOADSEARCH is just "../lisp".
+(let ((dir (car load-path)))
+  (message "load path is %S" load-path)
+  (setq load-path (list (expand-file-name "." dir)
+                        (expand-file-name "emacs-lisp" dir)
+                        (expand-file-name "language" dir)
+                        (expand-file-name "international" dir)
+                        (expand-file-name "textmodes" dir)
+                        (expand-file-name "vc" dir))))
+
+(setq purify-flag nil)
+
 (if (or (equal (member "bootstrap" command-line-args) '("bootstrap"))
 	;; FIXME this is irritatingly fragile.
 	(equal (nth 4 command-line-args) "unidata-gen.el")
@@ -64,19 +75,10 @@
 	(if (fboundp 'dump-emacs)
 	    (string-match "src/bootstrap-emacs" (nth 0 command-line-args))
 	  t))
-    (let ((dir (car load-path)))
-      ;; We'll probably overflow the pure space.
-      (setq purify-flag nil)
-      ;; Value of max-lisp-eval-depth when compiling initially.
-      ;; During bootstrapping the byte-compiler is run interpreted when
-      ;; compiling itself, which uses a lot more stack than usual.
-      (setq max-lisp-eval-depth 2200)
-      (setq load-path (list (expand-file-name "." dir)
-			    (expand-file-name "emacs-lisp" dir)
-			    (expand-file-name "language" dir)
-			    (expand-file-name "international" dir)
-			    (expand-file-name "textmodes" dir)
-			    (expand-file-name "vc" dir)))))
+    ;; Value of max-lisp-eval-depth when compiling initially.
+    ;; During bootstrapping the byte-compiler is run interpreted when
+    ;; compiling itself, which uses a lot more stack than usual.
+    (setq max-lisp-eval-depth 2200))
 
 (if (eq t purify-flag)
     ;; Hash consing saved around 11% of pure space in my tests.
@@ -308,6 +310,8 @@
 ;; Preload some constants and floating point functions.
 (load "emacs-lisp/float-sup")
 
+(load "emacs-lisp/cl-macs")
+
 (load "vc/vc-hooks")
 (load "vc/ediff-hook")
 (load "uniquify")

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Mon, 27 Mar 2017 22:27:26 -0400
> Cc: emacs-devel@gnu.org
> 
> > I applied your patch, and while dumping I get an error message:
> > 
> >  Dumping into dumped.elc...preparing...
> >  Dumping into dumped.elc...generating...
> >  Symbol's function definition is void: cl-labels
> > 
> > and dumped.elc is not re-created.  What did I miss?
> 
> Looks like I missed a “require” or “load” to pull in cl-macs.  Perhaps it’s loaded by something else in my build that’s platform-dependent (X11 vs Windows?) and isn’t in yours; I’m not sure. But it isn’t working for me to just load it explicitly without fixing up the load path too.  Perhaps I should’ve just defined a helper function instead of using cl-labels.
> 
> For now, try adding this patch. It bootstraps for me, and should get cl-labels defined.

This fixes the problem, and Emacs now starts OK, so the abbrevs issue
is also solved.

I think you should push all the changes you asked me to apply as
patches.

What is the roadmap ahead?  Are there any known issues left, before we
can consider this be a candidate for merging to master, and asking
people to test it in their routine workflows before we actually merge?

Thanks.

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

On Mar 31, 2017, at 02:57, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> This fixes the problem, and Emacs now starts OK, so the abbrevs issue
> is also solved.

Great!

> I think you should push all the changes you asked me to apply as
> patches.

Will do, probably this weekend.

> What is the roadmap ahead?  Are there any known issues left, before we
> can consider this be a candidate for merging to master, and asking
> people to test it in their routine workflows before we actually merge?
> 
> Thanks.

There are a number of issues on my list.  Some can be dealt with while people are experimenting, or even after merging.  Others may affect usability, like the current inability to report clearly and consistently if dumped.elc can’t be found.  I haven’t even tested doing “make install”; I always run Emacs from the build tree.  There are a few other minor bugs, like a few unprintable definitions not getting dumped, that it’d be nice to address; I’ll go back over my list and take a look.

Ken

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

On Mar 31, 2017, at 04:40, Ken Raeburn <raeburn@raeburn.org> wrote:

> 
> On Mar 31, 2017, at 02:57, Eli Zaretskii <eliz@gnu.org> wrote:
>> 
>> This fixes the problem, and Emacs now starts OK, so the abbrevs issue
>> is also solved.
> 
> Great!
> 
>> I think you should push all the changes you asked me to apply as
>> patches.
> 
> Will do, probably this weekend.

Looks like the abbrev change isn’t actually working right… I got the quoting wrong, so the abbrev tables are constructed as (mostly) proper abbrev tables, and in the right order, but the “:parent” properties are bad. Working on fixing it up….

Re: Skipping unexec via a big .elc file

[Alan Mackenzie]

Hello, Ken.

On Mon, Apr 03, 2017 at 12:15:29 -0400, Ken Raeburn wrote:

> On Mar 31, 2017, at 04:40, Ken Raeburn <raeburn@raeburn.org> wrote:


> > On Mar 31, 2017, at 02:57, Eli Zaretskii <eliz@gnu.org> wrote:

> >> This fixes the problem, and Emacs now starts OK, so the abbrevs issue
> >> is also solved.

> > Great!

> >> I think you should push all the changes you asked me to apply as
> >> patches.

> > Will do, probably this weekend.

> Looks like the abbrev change isn’t actually working right… I got the
> quoting wrong, so the abbrev tables are constructed as (mostly) proper
> abbrev tables, and in the right order, but the “:parent” properties are
> bad. Working on fixing it up….

I, for one, am feeling enthusiastic about this new way of building Emacs,
and am looking forward to trying it out in the near future.

Thanks for a great job almost finished!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

On Apr 3, 2017, at 12:57, Alan Mackenzie <acm@muc.de> wrote:

> Hello, Ken.
> 

> I, for one, am feeling enthusiastic about this new way of building Emacs,
> and am looking forward to trying it out in the near future.
> 
> Thanks for a great job almost finished!

Just making sure credit goes where it’s due:  Stefan did great work on the key piece, processing the Lisp environment for dumping as Lisp.  I’ve tried to improve the Lisp reader performance a bit, and fix up a couple minor bugs here and there, and maybe put a little polish on it.

Despite a few little speed-ups, I’ve got my doubts as to whether it’s going to be fast enough.  The .elc files are still (essentially) Lisp, and parsing text is not the most efficient way to load a bunch of object definitions.

Ken

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Mon, 3 Apr 2017 14:35:16 -0400
> Cc: emacs-devel@gnu.org
> 
> Despite a few little speed-ups, I’ve got my doubts as to whether it’s going to be fast enough.

I published my preliminary timings in these 2 messages:

  http://lists.gnu.org/archive/html/emacs-devel/2016-12/msg00923.html
  http://lists.gnu.org/archive/html/emacs-devel/2016-12/msg00959.html

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

On Apr 3, 2017, at 15:14, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Ken Raeburn <raeburn@raeburn.org>
>> Date: Mon, 3 Apr 2017 14:35:16 -0400
>> Cc: emacs-devel@gnu.org
>> 
>> Despite a few little speed-ups, I’ve got my doubts as to whether it’s going to be fast enough.
> 
> I published my preliminary timings in these 2 messages:
> 
>  http://lists.gnu.org/archive/html/emacs-devel/2016-12/msg00923.html
>  http://lists.gnu.org/archive/html/emacs-devel/2016-12/msg00959.html

Yes, I got some speedups, but I didn’t get it as fast as I was hoping.  Some of my changes since your second message above might’ve improved the numbers a little, but some (like loading the doc pointers at startup, and I think “uniquify” is going to need to be loaded at startup too because it attaches advice to “rename-buffer” which we can’t save properly) may slow it a little too.

I was aiming for a startup time under a tenth of a second, and didn’t get there, though there were a couple of additional things that could be tried, with some effort.  I’m not sure a startup time of nearly a fifth of a second will feel for people.  If they start Emacs once as part of logging in, it probably won’t be an issue.  If they start it every time they want to edit a file, it may be annoying to have the startup time increased by even 0.15s.

Still, I suppose we can let people try it out, and find out what they think.  Then we can decide if it’s good enough, if further speedup measures are worth exploring, or if it’s a dead end.

Ken

Re: Skipping unexec via a big .elc file

[Robert Pluim]

Ken Raeburn <raeburn@raeburn.org> writes:

>
> I was aiming for a startup time under a tenth of a second, and didn’t
> get there, though there were a couple of additional things that could
> be tried, with some effort.  I’m not sure a startup time of nearly a
> fifth of a second will feel for people.  If they start Emacs once as
> part of logging in, it probably won’t be an issue.  If they start it
> every time they want to edit a file, it may be annoying to have the
> startup time increased by even 0.15s.

I have an emacs I normally keep running, and occasionally start one
for a quick editing task. 0.15s is completely lost in the noise for
me. I've tried the branch, as far as I'm concerned its speed is fine.

Regards

Robert

Re: Skipping unexec via a big .elc file

[joakim]

Ken Raeburn <raeburn@raeburn.org> writes:

> On Apr 3, 2017, at 15:14, Eli Zaretskii <eliz@gnu.org> wrote:
>
>>> From: Ken Raeburn <raeburn@raeburn.org>
>>> Date: Mon, 3 Apr 2017 14:35:16 -0400
>>> Cc: emacs-devel@gnu.org
>>> 
>>> Despite a few little speed-ups, I’ve got my doubts as to whether it’s going to be fast enough.
>> 
>> I published my preliminary timings in these 2 messages:
>> 
>>  http://lists.gnu.org/archive/html/emacs-devel/2016-12/msg00923.html>  http://lists.gnu.org/archive/html/emacs-devel/2016-12/msg00959.html
>
> Yes, I got some speedups, but I didn’t get it as fast as I was hoping.  Some of my changes since your second message above might’ve improved the numbers a little, but some (like loading the doc pointers at startup, and I think “uniquify” is going to need to be loaded at startup too because it attaches advice to “rename-buffer” which we can’t save properly) may slow it a little too.
>
> I was aiming for a startup time under a tenth of a second, and didn’t get there, though there were a couple of additional things that could be tried, with some effort.  I’m not sure a startup time of nearly a fifth of a second will feel for people.  If they start Emacs once as part of logging in, it probably won’t be an issue.  If they start it every time they want to edit a file, it may be annoying to have the startup time increased by even 0.15s.

In my case I mostly use long-running sessions, so slow emacs startup
isn't so bad for me. Most of the boot time seem to happen in 3rd party
libs anyway.

But on the other hand I think there is a valid use case for using Emacs
for things like batch processing, web servers and such. And in those
cases startup time matters. Again otoh, you might want to use
emacsclient together with a long running emacs in those cases. But I'm
not really using emacs for that sort of thing so one should listen to
the people actually doing it primarily.

>
> Still, I suppose we can let people try it out, and find out what they think.  Then we can decide if it’s good enough, if further speedup measures are worth exploring, or if it’s a dead end.
>
> Ken
-- 
Joakim Verona
joakim@verona.se

Re: Skipping unexec via a big .elc file

[Clément Pit-Claudel]

On 2017-04-04 06:27, joakim@verona.se wrote:
> 
> But on the other hand I think there is a valid use case for using Emacs
> for things like batch processing, web servers and such. And in those
> cases startup time matters. Again otoh, you might want to use
> emacsclient together with a long running emacs in those cases. But I'm
> not really using emacs for that sort of thing so one should listen to
> the people actually doing it primarily.

I use Emacs in batch mode a lot; using a server is tricky, because anything you load or change on one run persists until the across future executions (for example one execution might load a file, then the next one might forget to load the file explicitly, but still work well because that file was previously loaded; when you run the program again in a fresh instance, things fail).

Additionally, there are bugs in Emacsclient that make it tricky to use on its own, so my current code relies on two instances of Emacs: a long-lived one and a short-lived on. THe short-lived one is used to connect to the long-running server. (This saves a lot because most of the execution time is otherwise spent loading packages).

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Tue, 4 Apr 2017 08:14:48 -0400
> 
> Additionally, there are bugs in Emacsclient that make it tricky to use on its own

Why aren't these bugs being fixed?  Are there bug reports with the
details?

Re: Skipping unexec via a big .elc file

[Clément Pit-Claudel]

On 2017-04-04 10:38, Eli Zaretskii wrote:
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
>> Date: Tue, 4 Apr 2017 08:14:48 -0400
>>
>> Additionally, there are bugs in Emacsclient that make it tricky to use on its own
> 
> Why aren't these bugs being fixed?  Are there bug reports with the
> details?

I can think of 3: "Is emacsclient --eval broken?" from emacs-devel on 2016-08-03, which got fixed almost instantly (thanks Johan Bockgård!). #24616, which now has documentation and so is arguably fixed (but errors still pop up on the server, not the client, and so the server has to capture backtraces and send them back explicitly). "How can I rethrow an error after recording a backtrace?" from emacs-devel on 2016-08-04, which is due to emacsclient not incrementing num_nonmacro_input_events.

One of them is fixed, the second seems seems to be mostly wontfix, and the third is open.  But all three are relevant when trying to remain compatible with older Emacsen.

Cheers,
Clément.

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Clément Pit-Claudel <cpitclaudel@gmail.com>
> Date: Tue, 4 Apr 2017 11:16:35 -0400
> Cc: emacs-devel@gnu.org
> 
> One of them is fixed, the second seems seems to be mostly wontfix, and the third is open.  But all three are relevant when trying to remain compatible with older Emacsen.

So there's only one bug that remains, is that right?

As for older versions, you could fix them by retrofitting the patches
into them, right?  And in any case, they seem to be unrelated to the
issue at hand, which will only affect the next release and those after
it.  Right?

Re: Skipping unexec via a big .elc file

[Clément Pit-Claudel]

On 2017-04-04 11:53, Eli Zaretskii wrote:
>> From: Clément Pit-Claudel <cpitclaudel@gmail.com> Date: Tue, 4 Apr
>> 2017 11:16:35 -0400 Cc: emacs-devel@gnu.org
>> 
>> One of them is fixed, the second seems seems to be mostly wontfix,
>> and the third is open.  But all three are relevant when trying to
>> remain compatible with older Emacsen.
> 
> So there's only one bug that remains, is that right?
> 
> As for older versions, you could fix them by retrofitting the
> patches into them, right?

Correct. When the patches are in Lisp, yes, to some extent.  In C, no, of course.

> And in any case, they seem to be unrelated to the issue at hand,
> which will only affect the next release and those after it.  Right?

Correct. I sought to point out that, since old version prevent some authors from using emacsclient, emacsclient being fast does mitigate much of the costs of Emacs itself starting slowly.

Re: Skipping unexec via a big .elc file

[Lars Brinkhoff]

Ken Raeburn wrote:
> I was aiming for a startup time under a tenth of a second, and didn’t
> get there, though there were a couple of additional things that could
> be tried, with some effort.  I’m not sure a startup time of nearly a
> fifth of a second will feel for people.

Every invokation of async-start launches a new emacs subprocess, doesn't
it?  So startup time would also affect uses of async.el.

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Lars Brinkhoff <lars@nocrew.org>
> Date: Fri, 07 Apr 2017 07:46:12 +0200
> 
> Ken Raeburn wrote:
> > I was aiming for a startup time under a tenth of a second, and didn’t
> > get there, though there were a couple of additional things that could
> > be tried, with some effort.  I’m not sure a startup time of nearly a
> > fifth of a second will feel for people.
> 
> Every invokation of async-start launches a new emacs subprocess, doesn't
> it?  So startup time would also affect uses of async.el.

Perhaps we could have a separate, much smaller dumped.elc for batch
invocations, to cater to these use cases.  Ken, does this make sense?

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

On Apr 7, 2017, at 03:28, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Lars Brinkhoff <lars@nocrew.org>
>> Date: Fri, 07 Apr 2017 07:46:12 +0200
>> 
>> Ken Raeburn wrote:
>>> I was aiming for a startup time under a tenth of a second, and didn’t
>>> get there, though there were a couple of additional things that could
>>> be tried, with some effort.  I’m not sure a startup time of nearly a
>>> fifth of a second will feel for people.
>> 
>> Every invokation of async-start launches a new emacs subprocess, doesn't
>> it?  So startup time would also affect uses of async.el.
> 
> Perhaps we could have a separate, much smaller dumped.elc for batch
> invocations, to cater to these use cases.  Ken, does this make sense?

We could do it, sure.  For example, stuff relating to window systems probably isn’t of much use in batch mode.  One question is, do we change such things to use autoload in case the user’s init file references their functions, or do we require that the user know to use “load” or “require”?

Maybe we can find file currently loaded that we could change over to autoloading in all cases, improving the interactive startup time too?

Ken

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Fri, 7 Apr 2017 05:02:30 -0400
> Cc: Lars Brinkhoff <lars@nocrew.org>,
>  emacs-devel@gnu.org
> 
> > Perhaps we could have a separate, much smaller dumped.elc for batch
> > invocations, to cater to these use cases.  Ken, does this make sense?
> 
> We could do it, sure.  For example, stuff relating to window systems probably isn’t of much use in batch mode.  One question is, do we change such things to use autoload in case the user’s init file references their functions, or do we require that the user know to use “load” or “require”?

I don't think I understand the question: -batch implies -Q, so the
user's init file is not relevant.

> Maybe we can find file currently loaded that we could change over to autoloading in all cases, improving the interactive startup time too?

Yes, making dumped.elc smaller by using autoload is another way of
slashing some load time.

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

On Apr 7, 2017, at 09:40, Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Ken Raeburn <raeburn@raeburn.org>
>> Date: Fri, 7 Apr 2017 05:02:30 -0400
>> Cc: Lars Brinkhoff <lars@nocrew.org>,
>> emacs-devel@gnu.org
>> 
>>> Perhaps we could have a separate, much smaller dumped.elc for batch
>>> invocations, to cater to these use cases.  Ken, does this make sense?
>> 
>> We could do it, sure.  For example, stuff relating to window systems probably isn’t of much use in batch mode.  One question is, do we change such things to use autoload in case the user’s init file references their functions, or do we require that the user know to use “load” or “require”?
> 
> I don't think I understand the question: -batch implies -Q, so the
> user's init file is not relevant.

Sorry, was too late at night I guess.  For batch mode, it’s code loaded via -l options that might have to add new explicit dependencies, if we don’t add autoloads for everything.  I would expect autoloads would be the direction we’d want to go….

Re: Skipping unexec via a big .elc file

[Clément Pit-Claudel]

On 2017-04-07 12:02, Ken Raeburn wrote:
> Sorry, was too late at night I guess.  For batch mode, it’s code
> loaded via -l options that might have to add new explicit
> dependencies, if we don’t add autoloads for everything.  I would
> expect autoloads would be the direction we’d want to go….

Removing some preloaded packages in favor of autoloads is probably a good idea.  We should be careful, though: as previously discussed, essentially everything in packages that were previously preloaded needs to be autoloaded now, since many packages don't (require) the preloaded features that they use. 

Re: Skipping unexec via a big .elc file

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 954 bytes --]

Clément Pit-Claudel <cpitclaudel@gmail.com> schrieb am Fr., 7. Apr. 2017 um
18:18 Uhr:

> On 2017-04-07 12:02, Ken Raeburn wrote:
> > Sorry, was too late at night I guess.  For batch mode, it’s code
> > loaded via -l options that might have to add new explicit
> > dependencies, if we don’t add autoloads for everything.  I would
> > expect autoloads would be the direction we’d want to go….
>
> Removing some preloaded packages in favor of autoloads is probably a good
> idea.  We should be careful, though: as previously discussed, essentially
> everything in packages that were previously preloaded needs to be
> autoloaded now, since many packages don't (require) the preloaded features
> that they use.
>
>
>
Doesn't that effectively just move most of the code to loaddefs.el, from
which it again has to be either preloaded or byte-compiled into the "big
.elc file"? Does this really bring measurable benefits nowadays?

[-- Attachment #2: Type: text/html, Size: 1422 bytes --]

Re: Skipping unexec via a big .elc file

[Clément Pit-Claudel]

On 2017-04-08 11:03, Philipp Stephani wrote:
> Clément Pit-Claudel <cpitclaudel@gmail.com schrieb:
>> … essentially everything in packages that were previously
>> preloaded needs to be autoloaded now, since many packages don't
>> (require) the preloaded features that they use.

> Doesn't that effectively just move most of the code to loaddefs.el,
> from which it again has to be either preloaded or byte-compiled into
> the "big .elc file"? Does this really bring measurable benefits
> nowadays?

(Sorry if I'm misunderstanding you)

I think the idea is that you can defer loading the implementation of a significant fraction of currently-preloaded functions, because many of these are currently unused.

So the intended saving is that currently-preloaded but uncommonly-used functions would not be dumped to the big-elc (their signatures, in the form an autoload, would be).  Packages that use them without (require)-ing the corresponding feature first would still work, but startup would be faster.

(I hope I got this right)

Re: Skipping unexec via a big .elc file

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 2332 bytes --]

Clément Pit-Claudel <cpitclaudel@gmail.com> schrieb am Sa., 8. Apr. 2017 um
17:15 Uhr:

> On 2017-04-08 11:03, Philipp Stephani wrote:
> > Clément Pit-Claudel <cpitclaudel@gmail.com schrieb:
> >> … essentially everything in packages that were previously
> >> preloaded needs to be autoloaded now, since many packages don't
> >> (require) the preloaded features that they use.
>
> > Doesn't that effectively just move most of the code to loaddefs.el,
> > from which it again has to be either preloaded or byte-compiled into
> > the "big .elc file"? Does this really bring measurable benefits
> > nowadays?
>
> (Sorry if I'm misunderstanding you)
>
> I think the idea is that you can defer loading the implementation of a
> significant fraction of currently-preloaded functions, because many of
> these are currently unused.
>
> So the intended saving is that currently-preloaded but uncommonly-used
> functions would not be dumped to the big-elc (their signatures, in the form
> an autoload, would be).  Packages that use them without (require)-ing the
> corresponding feature first would still work, but startup would be faster.
>

The question is whether there is actually a significant speed-up.
Autoloading is traditionally used for a small number of interactive
commands that cause large optional libraries to be loaded. In such cases I
could imagine that the performance gain is still significant. However, you
now suggest that preloaded libraries get turned into autoloads. The
structure of those libraries is typically quite different: the consist to a
large extent of individual helper functions that are independent of each
other. My guess is that this could make overall performance worse: it will
cause loaddefs.el to contain all the signatures and docstrings of these
helper functions, and loaddefs.el is itself not byte-compiled. Therefore,
you now need to load the definitions effectively twice: once in
loaddefs.el, once the functions are actually used. Therefore such a change
shouldn't be made without measuring its impact.
I'd actually prefer going into the other direction: preload much more than
now, and remove lots of stuff from autoloads. This will probably need a
different strategy for preloading (Daniel's approach, or Rmacs, or an Elisp
LLVM compiler, ...).

[-- Attachment #2: Type: text/html, Size: 3057 bytes --]

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Sat, 08 Apr 2017 15:53:49 +0000
> 
> The question is whether there is actually a significant speed-up.
> Autoloading is traditionally used for a small number of interactive commands that cause large optional libraries
> to be loaded. In such cases I could imagine that the performance gain is still significant. However, you now
> suggest that preloaded libraries get turned into autoloads. The structure of those libraries is typically quite
> different: the consist to a large extent of individual helper functions that are independent of each other. My
> guess is that this could make overall performance worse: it will cause loaddefs.el to contain all the signatures
> and docstrings of these helper functions, and loaddefs.el is itself not byte-compiled. Therefore, you now need
> to load the definitions effectively twice: once in loaddefs.el, once the functions are actually used. Therefore
> such a change shouldn't be made without measuring its impact.

This issue will not be resolved by guessing, but by measurements.  So
if you are interested and can produce a dumped.elc that only loads
what's necessary in -batch session, and that dumped.elc does or
doesn't load significantly faster than the full one, we will know who
is right here.

Thanks.

> I'd actually prefer going into the other direction: preload much more than now, and remove lots of stuff from
> autoloads. This will probably need a different strategy for preloading (Daniel's approach, or Rmacs, or an Elisp
> LLVM compiler, ...).

Given that load time is an issue, loading more stuff than strictly
necessary seems to make very little sense to me.

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

>> I'd actually prefer going into the other direction: preload much more than
>> now, and remove lots of stuff from
>> autoloads. This will probably need a different strategy for preloading
>> (Daniel's approach, or Rmacs, or an Elisp
>> LLVM compiler, ...).
> Given that load time is an issue, loading more stuff than strictly
> necessary seems to make very little sense to me.

IIUC, using Daniel's approach, it should be possible to preload using
mmap in a time that's largely independent from the size of the
preloaded file (or at least, with a small constant).


        Stefan

Re: Skipping unexec via a big .elc file

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 705 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> schrieb am Sa., 8. Apr. 2017 um
20:06 Uhr:

> >> I'd actually prefer going into the other direction: preload much more
> than
> >> now, and remove lots of stuff from
> >> autoloads. This will probably need a different strategy for preloading
> >> (Daniel's approach, or Rmacs, or an Elisp
> >> LLVM compiler, ...).
> > Given that load time is an issue, loading more stuff than strictly
> > necessary seems to make very little sense to me.
>
> IIUC, using Daniel's approach, it should be possible to preload using
> mmap in a time that's largely independent from the size of the
> preloaded file (or at least, with a small constant).
>
>
Yes, that would be ideal.

[-- Attachment #2: Type: text/html, Size: 1071 bytes --]

Re: Skipping unexec via a big .elc file

[Clément Pit-Claudel]

On 2017-04-08 11:53, Philipp Stephani wrote:
> However, you now suggest that preloaded libraries get turned into 
> autoloads.

I didn't suggest this :) I just pointed out that there were difficulties with that approach.

> Therefore such a change shouldn't be made without measuring its 
> impact.

Yup, as always when trying to optimize things :)

> I'd actually prefer going into the other direction: preload much more
> than now, and remove lots of stuff from autoloads. This will probably
> need a different strategy for preloading (Daniel's approach, or
> Rmacs, or an Elisp LLVM compiler, ...).

I don't have an opinion on this topic; I admire the work of both Daniel and Ken, but I'm happy to defer to you and other experts for technical opinions.

Clément.

Re: Skipping unexec via a big .elc file

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 699 bytes --]

Clément Pit-Claudel <cpitclaudel@gmail.com> schrieb am Sa., 8. Apr. 2017 um
19:58 Uhr:

>
> > I'd actually prefer going into the other direction: preload much more
> > than now, and remove lots of stuff from autoloads. This will probably
> > need a different strategy for preloading (Daniel's approach, or
> > Rmacs, or an Elisp LLVM compiler, ...).
>
> I don't have an opinion on this topic; I admire the work of both Daniel
> and Ken, but I'm happy to defer to you and other experts for technical
> opinions.
>
>
I'm absolutely not an expert on this. All I'm suggesting is that the impact
of such changes should be measured, and that startup time in batch mode
isn't everything.

[-- Attachment #2: Type: text/html, Size: 1018 bytes --]

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Mon, 01 May 2017 11:40:46 +0000
> 
> All I'm suggesting is that the impact of such changes should be
> measured, and that startup time in batch mode isn't everything. 

Startup time in batch mode isn't everything, but if it's a frequent
use case in which even relatively short delays are tangible, we should
try to find a way of minimizing those delays.

Re: Skipping unexec via a big .elc file

[Daniel Colascione]

On 05/01/2017 05:07 AM, Eli Zaretskii wrote:
>> From: Philipp Stephani <p.stephani2@gmail.com>
>> Date: Mon, 01 May 2017 11:40:46 +0000
>>
>> All I'm suggesting is that the impact of such changes should be
>> measured, and that startup time in batch mode isn't everything.
>
> Startup time in batch mode isn't everything, but if it's a frequent
> use case in which even relatively short delays are tangible, we should
> try to find a way of minimizing those delays.

I'm in a position to rebase my portable dumper patch. The last few 
months have been, er, interesting.

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Date: Thu, 18 May 2017 10:39:34 -0700
> Cc: cpitclaudel@gmail.com, emacs-devel@gnu.org
> 
> I'm in a position to rebase my portable dumper patch. The last few 
> months have been, er, interesting.

Please make a branch with the patch, so people could try it.

Thanks.

Re: Skipping unexec via a big .elc file

[Philipp Stephani]

Am Do., 18. Mai 2017 um 21:45 Uhr schrieb Eli Zaretskii <eliz@gnu.org>:
>
> > From: Daniel Colascione <dancol@dancol.org>
> > Date: Thu, 18 May 2017 10:39:34 -0700
> > Cc: cpitclaudel@gmail.com, emacs-devel@gnu.org
> >
> > I'm in a position to rebase my portable dumper patch. The last few
> > months have been, er, interesting.
>
> Please make a branch with the patch, so people could try it.

Hi, what's the state of the pdumper branch? Any chance we can merge it
to master?

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Philipp Stephani <p.stephani2@gmail.com>
> Date: Tue, 25 Dec 2018 16:46:23 +0100
> Cc: Daniel Colascione <dancol@dancol.org>, Clément Pit-Claudel <cpitclaudel@gmail.com>, 
> 	Emacs developers <emacs-devel@gnu.org>
> 
> Hi, what's the state of the pdumper branch? Any chance we can merge it
> to master?

Last time this popped up:

  http://lists.gnu.org/archive/html/emacs-devel/2018-10/msg00295.html

Re: Skipping unexec via a big .elc file

[Daniel Colascione]

>> From: Philipp Stephani <p.stephani2@gmail.com>
>> Date: Tue, 25 Dec 2018 16:46:23 +0100
>> Cc: Daniel Colascione <dancol@dancol.org>, Clément Pit-Claudel
>> <cpitclaudel@gmail.com>,
>> 	Emacs developers <emacs-devel@gnu.org>
>>
>> Hi, what's the state of the pdumper branch? Any chance we can merge it
>> to master?
>
> Last time this popped up:
>
>   http://lists.gnu.org/archive/html/emacs-devel/2018-10/msg00295.html
>

Yeah, it's about time we finally get around to doing this. I have some
time between now and the end of the year, and I'll rebase the work and
land it, assuming no new major objections.

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> Date: Tue, 25 Dec 2018 11:15:57 -0800
> From: "Daniel Colascione" <dancol@dancol.org>
> Cc: Philipp Stephani <p.stephani2@gmail.com>, dancol@dancol.org,
> 	cpitclaudel@gmail.com, emacs-devel@gnu.org
> 
> Yeah, it's about time we finally get around to doing this. I have some
> time between now and the end of the year, and I'll rebase the work and
> land it, assuming no new major objections.

Thanks.

Re: Skipping unexec via a big .elc file

[Daniel Colascione]

>>> From: Philipp Stephani <p.stephani2@gmail.com>
>>> Date: Tue, 25 Dec 2018 16:46:23 +0100
>>> Cc: Daniel Colascione <dancol@dancol.org>, Clément Pit-Claudel
>>> <cpitclaudel@gmail.com>,
>>> 	Emacs developers <emacs-devel@gnu.org>
>>>
>>> Hi, what's the state of the pdumper branch? Any chance we can merge it
>>> to master?
>>
>> Last time this popped up:
>>
>>   http://lists.gnu.org/archive/html/emacs-devel/2018-10/msg00295.html
>>
>
> Yeah, it's about time we finally get around to doing this. I have some
> time between now and the end of the year, and I'll rebase the work and
> land it, assuming no new major objections.

I'm still working on this, FWIW. I'd hoped it'd be a simple rebase, but
the vectorization of Lisp_Misc forced more changes than I'd thought.

Re: Skipping unexec via a big .elc file

[Daniel Colascione]

>>>> From: Philipp Stephani <p.stephani2@gmail.com>
>>>> Date: Tue, 25 Dec 2018 16:46:23 +0100
>>>> Cc: Daniel Colascione <dancol@dancol.org>, Clément Pit-Claudel
>>>> <cpitclaudel@gmail.com>,
>>>> 	Emacs developers <emacs-devel@gnu.org>
>>>>
>>>> Hi, what's the state of the pdumper branch? Any chance we can merge it
>>>> to master?
>>>
>>> Last time this popped up:
>>>
>>>   http://lists.gnu.org/archive/html/emacs-devel/2018-10/msg00295.html
>>>
>>
>> Yeah, it's about time we finally get around to doing this. I have some
>> time between now and the end of the year, and I'll rebase the work and
>> land it, assuming no new major objections.
>
> I'm still working on this, FWIW. I'd hoped it'd be a simple rebase, but
> the vectorization of Lisp_Misc forced more changes than I'd thought.

> Hi Daniel.
>> I'm still working on this, FWIW. I'd hoped it'd be a simple rebase, but
>> the vectorization of Lisp_Misc forced more changes than I'd thought.
>>
>
> Good to know knowing You are working on this.

I landed pdumper. It works on my machine (tm)! Let me know about any
breakage. Plenty of people tested the old pdumper branch, but the changes
necessary to rebase it could use a good once over.

Re: Skipping unexec via a big .elc file

[Tassilo Horn]

"Daniel Colascione" <dancol@dancol.org> writes:

> I landed pdumper. It works on my machine (tm)! Let me know about any
> breakage. Plenty of people tested the old pdumper branch, but the
> changes necessary to rebase it could use a good once over.

I'm not sure if it's related, but since today I encounter bug#34094.

Bye,
Tassilo

Re: Skipping unexec via a big .elc file

[Robert Pluim]

"Daniel Colascione" <dancol@dancol.org> writes:


> I landed pdumper. It works on my machine (tm)! Let me know about any
> breakage. Plenty of people tested the old pdumper branch, but the changes
> necessary to rebase it could use a good once over.

It works for me on x86_64-apple-darwin18.2.0 and x86_64-pc-linux-gnu
(I needed 'make bootstrap', which I guess is not unexpected for such a
big change).

Robert

Re: Skipping unexec via a big .elc file

[Phillip Lord]

I thought to build some windows snapshot binaries. I am getting this
error of master:


CC       pdumper.o
../../../../git/master/src/pdumper.c: In function 'dump_cold_bignum':
../../../../git/master/src/pdumper.c:3447:53: error: conversion to 'mp_size_t {aka long int}' from 'size_t {aka long long unsigned int}' may alter its value [-Werror=conversion]
       mp_limb_t limb = mpz_getlimbn (bignum->value, i);
                                                     ^
cc1.exe: some warnings being treated as errors
make[1]: *** [Makefile:392: pdumper.o] Error 1
make[1]: Leaving directory '/home/Administrator/emacs-build/build/emacs-27.0.50-snapshot/x86_64/src'
make: *** [Makefile:423: src] Error 2


Phil

Windows Binaries with pdumper

[Phillip Lord]

I've updated the snapshot binaries for Windows to the latest trunk,
which includes the pdumper.

https://alpha.gnu.org/gnu/emacs/pretest/windows/emacs-27/

Phil

Re: Windows Binaries with pdumper

[Jostein Kjønigsen]

[-- Attachment #1: Type: text/plain, Size: 605 bytes --]

Hey Phil.

Thanks for the build.That's great news!

I've installed it and seems to run just fine, while loading it with a
full config and doing some very basic tasks.
Are there any particular areas of this build you want tested?

--
Vennlig hilsen
Jostein Kjønigsen

jostein@kjonigsen.net 🍵 jostein@gmail.com
https://jostein.kjonigsen.net


On Fri, Jan 18, 2019, at 1:46 PM, Phillip Lord wrote:
> 
> I've updated the snapshot binaries for Windows to the latest trunk,
> which includes the pdumper.
> 
> https://alpha.gnu.org/gnu/emacs/pretest/windows/emacs-27/
> 
> Phil
> 


[-- Attachment #2: Type: text/html, Size: 1553 bytes --]

Re: Windows Binaries with pdumper

[Phillip Lord]

I am not the best person to ask this, since I don't use either windows
nor have much knowledge of the pdumper. The motivation for putting the
snapshots up was to get it into daily use by more people.

Phil

Jostein Kjønigsen <jostein@secure.kjonigsen.net> writes:

> Hey Phil.
>
> Thanks for the build.That's great news!
>
> I've installed it and seems to run just fine, while loading it with a
> full config and doing some very basic tasks.
> Are there any particular areas of this build you want tested?
>
> --
> Vennlig hilsen
> Jostein Kjønigsen
>
> jostein@kjonigsen.net 🍵 jostein@gmail.com
> https://jostein.kjonigsen.net
>
>
> On Fri, Jan 18, 2019, at 1:46 PM, Phillip Lord wrote:
>> 
>> I've updated the snapshot binaries for Windows to the latest trunk,
>> which includes the pdumper.
>> 
>> https://alpha.gnu.org/gnu/emacs/pretest/windows/emacs-27/
>> 
>> Phil
>> 

Re: Skipping unexec via a big .elc file

[Elias Mårtenson]

[-- Attachment #1: Type: text/plain, Size: 556 bytes --]

On Wed, 16 Jan 2019 at 06:47, Daniel Colascione <dancol@dancol.org> wrote:

I landed pdumper. It works on my machine (tm)! Let me know about any
> breakage. Plenty of people tested the old pdumper branch, but the changes
> necessary to rebase it could use a good once over.
>

Very nice, I've waited for this. Thanks a lot. I'm using it right now, and
it seems to work fine.

A small comment. There seems to be a typo in the word “build” in NEWS:

“Emacs now needs an "emacs.pdmp" file, generated during the built”.

Regards,
Elias

[-- Attachment #2: Type: text/html, Size: 1004 bytes --]

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> Date: Tue, 15 Jan 2019 14:46:03 -0800
> From: "Daniel Colascione" <dancol@dancol.org>
> Cc: "Daniel Colascione" <dancol@dancol.org>,
>  "Eli Zaretskii" <eliz@gnu.org>,
>  "Philipp Stephani" <p.stephani2@gmail.com>,
>  cpitclaudel@gmail.com,
>  emacs-devel@gnu.org
> 
> I landed pdumper.

Thanks.

Re: Skipping unexec via a big .elc file

[Daniel Colascione]

>> Date: Tue, 15 Jan 2019 14:46:03 -0800
>> From: "Daniel Colascione" <dancol@dancol.org>
>> Cc: "Daniel Colascione" <dancol@dancol.org>,
>>  "Eli Zaretskii" <eliz@gnu.org>,
>>  "Philipp Stephani" <p.stephani2@gmail.com>,
>>  cpitclaudel@gmail.com,
>>  emacs-devel@gnu.org
>>
>> I landed pdumper.
>
> Thanks.

Should have a fix for the crash in detect_coding soon-ish. Who knew that
struct coding_system had function pointers? (This structure is _mostly_
ephemeral, reloaded from Lisp, but it's also persistent on some contexts.)

Re: Skipping unexec via a big .elc file

[Clément Pit-Claudel]

On 15/01/2019 17.46, Daniel Colascione wrote:
> I landed pdumper.

Congratulations!

compiled lisp file format (Re: Skipping unexec via a big .elc file)

[Ken Raeburn]

I haven’t had much time to further the work on the big-elc approach recently, but there is one idea I want to toss out there for possibly improving the load time further: Changing the .elc file format to a binary one.  I’m not talking about a memory image like Daniel is working on.  I mean a file representing a sequence of S-expressions, but optimized for loading speed rather than for human readability.

The Guile project has taken this idea pretty far; they’re generating ELF object files with a few special sections for Guile objects, using the standard DWARF sections for debug information, etc.  While it has a certain appeal (making C modules and Lisp files look much more similar, maybe being able to link Lisp and C together into one executable image, letting GDB understand some of your data), switching to a machine-specific format would be a pretty drastic change, when we can currently share the files across machines.

I haven’t got a complete, concrete proposal, but I see at least a couple general approaches possible:

1) Follow the model of flat object file formats: Some file sections have data of various types (string content, symbol names, integer or floating constants); others (the equivalent of standard object file “relocation” data) would provide info on how to allocate and fill in the container objects (pairs, vectors, etc) desired, with references to the symbols or strings or other container objects.

2) Continue to use the current recursive processing, but with a binary format.  Some (byte? word?) value indicates “this is string data”, it’s followed by a byte count and that many bytes of string content (always using the Emacs internal encoding, so we don’t have to translate when reading).  Another value indicates an integer constant.  Another value indicates a vector, and is followed by a length and then that many other values, which are each processed recursively before we get back to the object following the vector.  Each object’s initializer’s length is dependent on the type, and for container types, the values contained within.

Either way, getting away from the expensive one-character-at-a-time processing, multibyte coding, escape processing, etc., and pushing around groups of bytes whenever possible should save us time.

This would be useable not just for the dumped.elc file, but for other compiled Lisp files as well, whether in the distribution or from ELPA or the user’s own code.

I did throw together a half-baked attempt to try some of this out.  I added a new “#” construct for unibyte strings, putting the byte count into the file so that the string data could be copied with fread() instead of a READCHAR loop.  I also added a new version of the “#n#” syntax that uses a fixed number of READCHAR calls and avoids the decimal arithmetic.  So, the file can no longer be processed as Lisp, and it still requires some text parsing, though not nearly as much as before; some of the worst of both worlds.  But the load time for dumped.elc did drop by another 12% in my tests (start in batch mode, print a message and exit, from 0.227s down to 0.2s or less per run, still loading a couple of standard-elc-format files during startup).

I’m curious if people think this might be an approach worth pursuing.  Or if the Lisp-based elc format is seen as advantageous in ways I’m not seeing….

Ken

Re: compiled lisp file format (Re: Skipping unexec via a big .elc file)

[Paul Eggert]

Ken Raeburn wrote:
> The Guile project has taken this idea pretty far; they’re generating ELF object files with a few special sections for Guile objects, using the standard DWARF sections for debug information, etc.  While it has a certain appeal (making C modules and Lisp files look much more similar, maybe being able to link Lisp and C together into one executable image, letting GDB understand some of your data), switching to a machine-specific format would be a pretty drastic change, when we can currently share the files across machines.

Although it does indeed sound like a big change, I don't see why it would 
prevent us from sharing the files across machines. Emacs can use standard ELF 
and DWARF format on any platform if Emacs is doing the loading. And there should 
be some software-engineering benefit in using the same format that Guile uses.

Re: compiled lisp file format (Re: Skipping unexec via a big .elc file)

[Ken Raeburn]

On May 21, 2017, at 04:53, Paul Eggert <eggert@cs.ucla.edu> wrote:

> Ken Raeburn wrote:
>> The Guile project has taken this idea pretty far; they’re generating ELF object files with a few special sections for Guile objects, using the standard DWARF sections for debug information, etc.  While it has a certain appeal (making C modules and Lisp files look much more similar, maybe being able to link Lisp and C together into one executable image, letting GDB understand some of your data), switching to a machine-specific format would be a pretty drastic change, when we can currently share the files across machines.
> 
> Although it does indeed sound like a big change, I don't see why it would prevent us from sharing the files across machines. Emacs can use standard ELF and DWARF format on any platform if Emacs is doing the loading. And there should be some software-engineering benefit in using the same format that Guile uses.

Sorry for the delay in responding.

The ELF format has header fields indicating the word size, endianness, machine architecture (though there’s a value for “none”), and OS ABI.  Some fields vary in size or order depending on whether the 32-bit or 64-bit format is in use.  Some other format details (e.g., relocation types, interpretation of certain ranges of values in some fields) are architecture- or OS-dependent; we might not care about many of those details, but relocations are likely needed if we want to play linking games or use DWARF.

I think Guile is using whatever the native word size and architecture are.  If we do that for Emacs, they’re not portable between platforms.  Currently it works for me to put my Lisp files, both source and compiled, into ~/elisp and use them from different kinds of machines if my home directory is NFS-mounted.

We could instead pick fixed values (say, architecture “none”, little-endian, 32-bit), but then there’s no guarantee that we could use any of the usual GNU tools on them without a bunch of work, or that we’d ever be able to use non-GNU tools to treat them as object files.  Then again, we couldn’t expect to do the latter portably anyway, since some of the platforms don’t even use ELF.

Ken

Re: compiled lisp file format (Re: Skipping unexec via a big .elc file)

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 2561 bytes --]

Ken Raeburn <raeburn@raeburn.org> schrieb am So., 28. Mai 2017 um 13:07 Uhr:

>
> On May 21, 2017, at 04:53, Paul Eggert <eggert@cs.ucla.edu> wrote:
>
> > Ken Raeburn wrote:
> >> The Guile project has taken this idea pretty far; they’re generating
> ELF object files with a few special sections for Guile objects, using the
> standard DWARF sections for debug information, etc.  While it has a certain
> appeal (making C modules and Lisp files look much more similar, maybe being
> able to link Lisp and C together into one executable image, letting GDB
> understand some of your data), switching to a machine-specific format would
> be a pretty drastic change, when we can currently share the files across
> machines.
> >
> > Although it does indeed sound like a big change, I don't see why it
> would prevent us from sharing the files across machines. Emacs can use
> standard ELF and DWARF format on any platform if Emacs is doing the
> loading. And there should be some software-engineering benefit in using the
> same format that Guile uses.
>
> Sorry for the delay in responding.
>
> The ELF format has header fields indicating the word size, endianness,
> machine architecture (though there’s a value for “none”), and OS ABI.  Some
> fields vary in size or order depending on whether the 32-bit or 64-bit
> format is in use.  Some other format details (e.g., relocation types,
> interpretation of certain ranges of values in some fields) are
> architecture- or OS-dependent; we might not care about many of those
> details, but relocations are likely needed if we want to play linking games
> or use DWARF.
>
> I think Guile is using whatever the native word size and architecture
> are.  If we do that for Emacs, they’re not portable between platforms.
> Currently it works for me to put my Lisp files, both source and compiled,
> into ~/elisp and use them from different kinds of machines if my home
> directory is NFS-mounted.
>
> We could instead pick fixed values (say, architecture “none”,
> little-endian, 32-bit), but then there’s no guarantee that we could use any
> of the usual GNU tools on them without a bunch of work, or that we’d ever
> be able to use non-GNU tools to treat them as object files.  Then again, we
> couldn’t expect to do the latter portably anyway, since some of the
> platforms don’t even use ELF.
>
>
Is there any significant advantage of using ELF, or could this just use one
of the standard binary serialization formats (protobuf, flatbuffer, ...)?

[-- Attachment #2: Type: text/html, Size: 2873 bytes --]

Re: compiled lisp file format (Re: Skipping unexec via a big .elc file)

[Ken Raeburn]

[-- Attachment #1: Type: text/plain, Size: 6033 bytes --]


On May 28, 2017, at 08:43, Philipp Stephani <p.stephani2@gmail.com> wrote:

> 
> 
> Ken Raeburn <raeburn@raeburn.org <mailto:raeburn@raeburn.org>> schrieb am So., 28. Mai 2017 um 13:07 Uhr:
> 
> On May 21, 2017, at 04:53, Paul Eggert <eggert@cs.ucla.edu <mailto:eggert@cs.ucla.edu>> wrote:
> 
> > Ken Raeburn wrote:
> >> The Guile project has taken this idea pretty far; they’re generating ELF object files with a few special sections for Guile objects, using the standard DWARF sections for debug information, etc.  While it has a certain appeal (making C modules and Lisp files look much more similar, maybe being able to link Lisp and C together into one executable image, letting GDB understand some of your data), switching to a machine-specific format would be a pretty drastic change, when we can currently share the files across machines.
> >
> > Although it does indeed sound like a big change, I don't see why it would prevent us from sharing the files across machines. Emacs can use standard ELF and DWARF format on any platform if Emacs is doing the loading. And there should be some software-engineering benefit in using the same format that Guile uses.
> 
> Sorry for the delay in responding.
> 
> The ELF format has header fields indicating the word size, endianness, machine architecture (though there’s a value for “none”), and OS ABI.  Some fields vary in size or order depending on whether the 32-bit or 64-bit format is in use.  Some other format details (e.g., relocation types, interpretation of certain ranges of values in some fields) are architecture- or OS-dependent; we might not care about many of those details, but relocations are likely needed if we want to play linking games or use DWARF.
> 
> I think Guile is using whatever the native word size and architecture are.  If we do that for Emacs, they’re not portable between platforms.  Currently it works for me to put my Lisp files, both source and compiled, into ~/elisp and use them from different kinds of machines if my home directory is NFS-mounted.
> 
> We could instead pick fixed values (say, architecture “none”, little-endian, 32-bit), but then there’s no guarantee that we could use any of the usual GNU tools on them without a bunch of work, or that we’d ever be able to use non-GNU tools to treat them as object files.  Then again, we couldn’t expect to do the latter portably anyway, since some of the platforms don’t even use ELF.
> 
> 
> Is there any significant advantage of using ELF, or could this just use one of the standard binary serialization formats (protobuf, flatbuffer, ...)? 

That’s an interesting idea.  If one of the popular serialization libraries is compatibly licensed, easy to use, and performs well, it may be better than rolling our own.  It’ll need to handle data structures with circular or cross-linked references.  And we have the doc string delayed-loading optimization (that currently uses #$ and #@ syntaxes); presumably we’d like to keep that optimization in some form.  It would be good not to have to build all our data structures on ones generated by the tool with its own bookkeeping fields; having anything in a cons cell besides the “car” and “cdr” slots would mean a significant increase in memory use.


I initially said, “follow the model of flat object file formats”, not “use ELF”; ELF is just one way of organizing the data of an object file, with years of experience behind it, which we could use wholesale or borrow some lessons from.  One of the typical advantages of object file formats is that the data is grouped for efficient memory usage; some sections of a file will be mapped into the address space read-only (shared between processes), other sections read-write (possibly shared until copied on write), and others not mapped at all.  For example, we might put symbol names (normally never modified but it can be done), doc strings (to be loaded later, only if needed), byte code, and other strings into their own sections, and create Lisp_String objects and such pointing to those bytes as needed.  We don’t keep much in the way of source location information for Lisp code around, but if we ever change that, arguably it could go in a file section that’s not mapped or read until the debugger wants the information.


The Guile project’s documentation says their use of ELF is intended to build on existing work to invent a good object file format with several desired characteristics (https://www.gnu.org/software/guile/manual/html_node/Object-File-Format.html):

	• Above all else, it should be very cheap to load a compiled file.
	• It should be possible to statically allocate constants in the file. For example, a bytevector literal in source code can be emitted directly into the object file.
	• The compiled file should enable maximum code and data sharing between different processes.
	• The compiled file should contain debugging information, such as line numbers, but that information should be separated from the code itself. It should be possible to strip debugging information if space is tight.

They’re generating byte code currently, but are looking forward towards generating native code as well (instead?).

Their write-up implicitly assumes that, as with “normal” object files, the idea is to mmap the data into the address space, some of it read-only and some of it automatically getting some patching up, and then using those in-memory objects directly.  There’s no explicit discussion of the tradeoffs of loading a file all at once versus reading one object tree (S-expression) at a time from an input stream, but especially when mapping and using much of the data unmodified is feasible, I suspect the all-at-once approach is likely to be more efficient.  Whether that would be true in a case like Emacs, I don’t know.

They use DWARF for carrying some debug information, but so far I’m unsure what information is actually stored there.

Ken

[-- Attachment #2: Type: text/html, Size: 8822 bytes --]

Re: compiled lisp file format (Re: Skipping unexec via a big .elc file)

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 3249 bytes --]

Ken Raeburn <raeburn@raeburn.org> schrieb am Mo., 29. Mai 2017 um 11:33 Uhr:

>
> On May 28, 2017, at 08:43, Philipp Stephani <p.stephani2@gmail.com> wrote:
>
>
>
> Ken Raeburn <raeburn@raeburn.org> schrieb am So., 28. Mai 2017 um
> 13:07 Uhr:
>
>>
>> On May 21, 2017, at 04:53, Paul Eggert <eggert@cs.ucla.edu> wrote:
>>
>> > Ken Raeburn wrote:
>> >> The Guile project has taken this idea pretty far; they’re generating
>> ELF object files with a few special sections for Guile objects, using the
>> standard DWARF sections for debug information, etc.  While it has a certain
>> appeal (making C modules and Lisp files look much more similar, maybe being
>> able to link Lisp and C together into one executable image, letting GDB
>> understand some of your data), switching to a machine-specific format would
>> be a pretty drastic change, when we can currently share the files across
>> machines.
>> >
>> > Although it does indeed sound like a big change, I don't see why it
>> would prevent us from sharing the files across machines. Emacs can use
>> standard ELF and DWARF format on any platform if Emacs is doing the
>> loading. And there should be some software-engineering benefit in using the
>> same format that Guile uses.
>>
>> Sorry for the delay in responding.
>>
>> The ELF format has header fields indicating the word size, endianness,
>> machine architecture (though there’s a value for “none”), and OS ABI.  Some
>> fields vary in size or order depending on whether the 32-bit or 64-bit
>> format is in use.  Some other format details (e.g., relocation types,
>> interpretation of certain ranges of values in some fields) are
>> architecture- or OS-dependent; we might not care about many of those
>> details, but relocations are likely needed if we want to play linking games
>> or use DWARF.
>>
>> I think Guile is using whatever the native word size and architecture
>> are.  If we do that for Emacs, they’re not portable between platforms.
>> Currently it works for me to put my Lisp files, both source and compiled,
>> into ~/elisp and use them from different kinds of machines if my home
>> directory is NFS-mounted.
>>
>> We could instead pick fixed values (say, architecture “none”,
>> little-endian, 32-bit), but then there’s no guarantee that we could use any
>> of the usual GNU tools on them without a bunch of work, or that we’d ever
>> be able to use non-GNU tools to treat them as object files.  Then again, we
>> couldn’t expect to do the latter portably anyway, since some of the
>> platforms don’t even use ELF.
>>
>>
> Is there any significant advantage of using ELF, or could this just use
> one of the standard binary serialization formats (protobuf, flatbuffer,
> ...)?
>
>
> That’s an interesting idea.  If one of the popular serialization libraries
> is compatibly licensed, easy to use, and performs well, it may be better
> than rolling our own.
>

I've tried this out (with flatbuffers), but I haven't seen significant
speed improvements. It might very well be the case that during loading the
reader is already fast enough (e.g. for ELC files it doesn't do any
decoding), and it's the evaluator that's too slow.

[-- Attachment #2: Type: text/html, Size: 4253 bytes --]

Re: compiled lisp file format (Re: Skipping unexec via a big .elc file)

[Ken Raeburn]

[-- Attachment #1: Type: text/plain, Size: 3869 bytes --]


On Jul 2, 2017, at 11:46, Philipp Stephani <p.stephani2@gmail.com> wrote:

> Ken Raeburn <raeburn@raeburn.org <mailto:raeburn@raeburn.org>> schrieb am Mo., 29. Mai 2017 um 11:33 Uhr:
> 
> On May 28, 2017, at 08:43, Philipp Stephani <p.stephani2@gmail.com <mailto:p.stephani2@gmail.com>> wrote:
> 
>> 
>> 
>> Ken Raeburn <raeburn@raeburn.org <mailto:raeburn@raeburn.org>> schrieb am So., 28. Mai 2017 um 13:07 Uhr:
>> 
>> On May 21, 2017, at 04:53, Paul Eggert <eggert@cs.ucla.edu <mailto:eggert@cs.ucla.edu>> wrote:
>> 
>> > Ken Raeburn wrote:
>> >> The Guile project has taken this idea pretty far; they’re generating ELF object files with a few special sections for Guile objects, using the standard DWARF sections for debug information, etc.  While it has a certain appeal (making C modules and Lisp files look much more similar, maybe being able to link Lisp and C together into one executable image, letting GDB understand some of your data), switching to a machine-specific format would be a pretty drastic change, when we can currently share the files across machines.
>> >
>> > Although it does indeed sound like a big change, I don't see why it would prevent us from sharing the files across machines. Emacs can use standard ELF and DWARF format on any platform if Emacs is doing the loading. And there should be some software-engineering benefit in using the same format that Guile uses.
>> 
>> Sorry for the delay in responding.
>> 
>> The ELF format has header fields indicating the word size, endianness, machine architecture (though there’s a value for “none”), and OS ABI.  Some fields vary in size or order depending on whether the 32-bit or 64-bit format is in use.  Some other format details (e.g., relocation types, interpretation of certain ranges of values in some fields) are architecture- or OS-dependent; we might not care about many of those details, but relocations are likely needed if we want to play linking games or use DWARF.
>> 
>> I think Guile is using whatever the native word size and architecture are.  If we do that for Emacs, they’re not portable between platforms.  Currently it works for me to put my Lisp files, both source and compiled, into ~/elisp and use them from different kinds of machines if my home directory is NFS-mounted.
>> 
>> We could instead pick fixed values (say, architecture “none”, little-endian, 32-bit), but then there’s no guarantee that we could use any of the usual GNU tools on them without a bunch of work, or that we’d ever be able to use non-GNU tools to treat them as object files.  Then again, we couldn’t expect to do the latter portably anyway, since some of the platforms don’t even use ELF.
>> 
>> 
>> Is there any significant advantage of using ELF, or could this just use one of the standard binary serialization formats (protobuf, flatbuffer, ...)? 
> 
> That’s an interesting idea.  If one of the popular serialization libraries is compatibly licensed, easy to use, and performs well, it may be better than rolling our own.
> 
> I've tried this out (with flatbuffers), but I haven't seen significant speed improvements. It might very well be the case that during loading the reader is already fast enough (e.g. for ELC files it doesn't do any decoding), and it's the evaluator that's too slow.

What’s your test case, and how are you measuring the performance?

In my tests with the one big elc file, using the Linux “perf” tool, it seems that readchar, read1, encode_char, and ungetc are where a good chunk of CPU time is still spent — about 1/4 in my testing with the “big elc file” code.  My experiment in May cut down a chunk of the overall run time (start in batch mode, print a message, and exit) with some ugly reader syntax hacks. Tests with smaller files may have different characteristics though…

Ken

[-- Attachment #2: Type: text/html, Size: 6224 bytes --]

Re: compiled lisp file format (Re: Skipping unexec via a big .elc file)

[Philipp Stephani]

[-- Attachment #1: Type: text/plain, Size: 3926 bytes --]

Ken Raeburn <raeburn@raeburn.org> schrieb am Mo., 3. Juli 2017 um 03:44 Uhr:

>
> On Jul 2, 2017, at 11:46, Philipp Stephani <p.stephani2@gmail.com> wrote:
>
> Ken Raeburn <raeburn@raeburn.org> schrieb am Mo., 29. Mai 2017 um
> 11:33 Uhr:
>
>>
>> On May 28, 2017, at 08:43, Philipp Stephani <p.stephani2@gmail.com>
>> wrote:
>>
>>
>>
>> Ken Raeburn <raeburn@raeburn.org> schrieb am So., 28. Mai 2017 um
>> 13:07 Uhr:
>>
>>>
>>> On May 21, 2017, at 04:53, Paul Eggert <eggert@cs.ucla.edu> wrote:
>>>
>>> > Ken Raeburn wrote:
>>> >> The Guile project has taken this idea pretty far; they’re generating
>>> ELF object files with a few special sections for Guile objects, using the
>>> standard DWARF sections for debug information, etc.  While it has a certain
>>> appeal (making C modules and Lisp files look much more similar, maybe being
>>> able to link Lisp and C together into one executable image, letting GDB
>>> understand some of your data), switching to a machine-specific format would
>>> be a pretty drastic change, when we can currently share the files across
>>> machines.
>>> >
>>> > Although it does indeed sound like a big change, I don't see why it
>>> would prevent us from sharing the files across machines. Emacs can use
>>> standard ELF and DWARF format on any platform if Emacs is doing the
>>> loading. And there should be some software-engineering benefit in using the
>>> same format that Guile uses.
>>>
>>> Sorry for the delay in responding.
>>>
>>> The ELF format has header fields indicating the word size, endianness,
>>> machine architecture (though there’s a value for “none”), and OS ABI.  Some
>>> fields vary in size or order depending on whether the 32-bit or 64-bit
>>> format is in use.  Some other format details (e.g., relocation types,
>>> interpretation of certain ranges of values in some fields) are
>>> architecture- or OS-dependent; we might not care about many of those
>>> details, but relocations are likely needed if we want to play linking games
>>> or use DWARF.
>>>
>>> I think Guile is using whatever the native word size and architecture
>>> are.  If we do that for Emacs, they’re not portable between platforms.
>>> Currently it works for me to put my Lisp files, both source and compiled,
>>> into ~/elisp and use them from different kinds of machines if my home
>>> directory is NFS-mounted.
>>>
>>> We could instead pick fixed values (say, architecture “none”,
>>> little-endian, 32-bit), but then there’s no guarantee that we could use any
>>> of the usual GNU tools on them without a bunch of work, or that we’d ever
>>> be able to use non-GNU tools to treat them as object files.  Then again, we
>>> couldn’t expect to do the latter portably anyway, since some of the
>>> platforms don’t even use ELF.
>>>
>>>
>> Is there any significant advantage of using ELF, or could this just use
>> one of the standard binary serialization formats (protobuf, flatbuffer,
>> ...)?
>>
>>
>> That’s an interesting idea.  If one of the popular serialization
>> libraries is compatibly licensed, easy to use, and performs well, it may
>> be better than rolling our own.
>>
>
> I've tried this out (with flatbuffers), but I haven't seen significant
> speed improvements. It might very well be the case that during loading the
> reader is already fast enough (e.g. for ELC files it doesn't do any
> decoding), and it's the evaluator that's too slow.
>
>
> What’s your test case, and how are you measuring the performance?
>

IIRC I've repeatedly loaded one of the biggest .elc files shipped with
Emacs and measured the total loading time. I haven't done any detailed
profiling, since I was hoping for a significant speed increase that would
justify the work.
If people are generally interested in pursuing this further, I'd be happy
to put my code into a scratch branch.

[-- Attachment #2: Type: text/html, Size: 5694 bytes --]

Re: compiled lisp file format (Re: Skipping unexec via a big .elc file)

[Ken Raeburn]

[-- Attachment #1: Type: text/plain, Size: 6615 bytes --]


On Sep 24, 2017, at 09:57, Philipp Stephani <p.stephani2@gmail.com> wrote:
> Ken Raeburn <raeburn@raeburn.org <mailto:raeburn@raeburn.org>> schrieb am Mo., 3. Juli 2017 um 03:44 Uhr:
> 
> On Jul 2, 2017, at 11:46, Philipp Stephani <p.stephani2@gmail.com <mailto:p.stephani2@gmail.com>> wrote:
> 
>> Ken Raeburn <raeburn@raeburn.org <mailto:raeburn@raeburn.org>> schrieb am Mo., 29. Mai 2017 um 11:33 Uhr:
>> 
>> On May 28, 2017, at 08:43, Philipp Stephani <p.stephani2@gmail.com <mailto:p.stephani2@gmail.com>> wrote:
>> 
>>> 
>>> 
>>> Ken Raeburn <raeburn@raeburn.org <mailto:raeburn@raeburn.org>> schrieb am So., 28. Mai 2017 um 13:07 Uhr:
>>> 
>>> On May 21, 2017, at 04:53, Paul Eggert <eggert@cs.ucla.edu <mailto:eggert@cs.ucla.edu>> wrote:
>>> 
>>> > Ken Raeburn wrote:
>>> >> The Guile project has taken this idea pretty far; they’re generating ELF object files with a few special sections for Guile objects, using the standard DWARF sections for debug information, etc.  While it has a certain appeal (making C modules and Lisp files look much more similar, maybe being able to link Lisp and C together into one executable image, letting GDB understand some of your data), switching to a machine-specific format would be a pretty drastic change, when we can currently share the files across machines.
>>> >
>>> > Although it does indeed sound like a big change, I don't see why it would prevent us from sharing the files across machines. Emacs can use standard ELF and DWARF format on any platform if Emacs is doing the loading. And there should be some software-engineering benefit in using the same format that Guile uses.
>>> 
>>> Sorry for the delay in responding.
>>> 
>>> The ELF format has header fields indicating the word size, endianness, machine architecture (though there’s a value for “none”), and OS ABI.  Some fields vary in size or order depending on whether the 32-bit or 64-bit format is in use.  Some other format details (e.g., relocation types, interpretation of certain ranges of values in some fields) are architecture- or OS-dependent; we might not care about many of those details, but relocations are likely needed if we want to play linking games or use DWARF.
>>> 
>>> I think Guile is using whatever the native word size and architecture are.  If we do that for Emacs, they’re not portable between platforms.  Currently it works for me to put my Lisp files, both source and compiled, into ~/elisp and use them from different kinds of machines if my home directory is NFS-mounted.
>>> 
>>> We could instead pick fixed values (say, architecture “none”, little-endian, 32-bit), but then there’s no guarantee that we could use any of the usual GNU tools on them without a bunch of work, or that we’d ever be able to use non-GNU tools to treat them as object files.  Then again, we couldn’t expect to do the latter portably anyway, since some of the platforms don’t even use ELF.
>>> 
>>> 
>>> Is there any significant advantage of using ELF, or could this just use one of the standard binary serialization formats (protobuf, flatbuffer, ...)? 
>> 
>> That’s an interesting idea.  If one of the popular serialization libraries is compatibly licensed, easy to use, and performs well, it may be better than rolling our own.
>> 
>> I've tried this out (with flatbuffers), but I haven't seen significant speed improvements. It might very well be the case that during loading the reader is already fast enough (e.g. for ELC files it doesn't do any decoding), and it's the evaluator that's too slow.
> 
> What’s your test case, and how are you measuring the performance?
> 
> IIRC I've repeatedly loaded one of the biggest .elc files shipped with Emacs and measured the total loading time. I haven't done any detailed profiling, since I was hoping for a significant speed increase that would justify the work.

It’ll depend on what the code in that file is doing.

In the raeburn-startup branch, the last bit of profiling I did — you can see a graph at http://www.mit.edu/~raeburn/emacs.svg <http://www.mit.edu/~raeburn/emacs.svg> and if you haven’t read up on flame graphs (http://www.brendangregg.com/flamegraphs.html <http://www.brendangregg.com/flamegraphs.html>), they provide a nice visualization of the CPU time consumption broken down by what the current call stack looks like — showed nearly 1/3 of the CPU time of a simple run of Emacs in batch mode was spent reading and parsing the saved Lisp environment.  Most of the rest of the CPU time was spent executing the loaded code (lots of fset and setplist calls), but the biggest chunk of that was executing a nested load of international/characters.elc; during that nested load, most of the time was spent in execution (mostly char table processing) and very little in parsing.

So… for the saved Lisp environment file, excluding the nested load, reading and parsing is about 2/3 of the CPU time used; for characters.elc, reading and parsing is a minuscule portion of the CPU time.

Loading a Lisp file internally uses the Lisp “read” routine, which requires an input stream of character values (not byte values) to be supplied; we examine the stream object and dispatch to various bits of code depending on its type (buffer, marker, function, certain special symbols), *for each character*.  Each byte is examined to see if it’s part of a multibyte character.  Each character is considered to see if it’s allowed to be part of a symbol name or string or whatever we’re in the middle of parsing, or if it’s a backslash quoting some other character, etc.

Hence my hopes for a non-text-based format, designed to streamline reading data from files, where we can do things like specify a vector length or string length up front instead of having to consider each character and process character quoting sequences, stuff like that.  E.g., here’s a unibyte string of 47 bytes, so just copy the bytes without considering every one separately.  No human-readable printed form, no escape sequences needed.

Another help might be finding a faster way to load the character data.  I’ve got the branch loading characters.elc at startup because saving and parsing the generated tables was even slower than evaluating the Lisp code to generate them.  Perhaps we can do some processing of them during the build and convert them into some other form that lets us start up faster.

> If people are generally interested in pursuing this further, I'd be happy to put my code into a scratch branch.

I’d be curious to take a look…

Ken

[-- Attachment #2: Type: text/html, Size: 9862 bytes --]

Re: compiled lisp file format (Re: Skipping unexec via a big .elc file)

[Paul Eggert]

Ken Raeburn wrote:
> I think Guile is using whatever the native word size and architecture are.  If we do that for Emacs, they’re not portable between platforms.

Sure, but we're talking about the format Emacs uses to save its state, not the 
format of .elc files. Currently Emacs saves its state as an executable file that 
in general cannot be moved from one GNU/Linux distribution to another even if 
they have the same architecture. Switching to Guile's platform-neutral approach 
would make Emacs's saved-state format more portable, not less.

Re: compiled lisp file format (Re: Skipping unexec via a big .elc file)

[Ken Raeburn]

On May 28, 2017, at 17:09, Paul Eggert <eggert@cs.ucla.edu> wrote:

> Ken Raeburn wrote:
>> I think Guile is using whatever the native word size and architecture are.  If we do that for Emacs, they’re not portable between platforms.
> 
> Sure, but we're talking about the format Emacs uses to save its state, not the format of .elc files. Currently Emacs saves its state as an executable file that in general cannot be moved from one GNU/Linux distribution to another even if they have the same architecture. Switching to Guile's platform-neutral approach would make Emacs's saved-state format more portable, not less.

Actually, I was referring to compiled-Lisp files generally, including the “dumped.elc” file, when I suggested it.

And I wouldn’t describe Guile’s “ELF everywhere” approach as entirely platform-neutral.  I built a Guile tree tonight to take a look.  My guess earlier about using the native architecture was wrong (it uses “none”), but it appears that the generated files are specific to the host’s byte order and word size.  So some sharing is possible between similar platforms, but not across all as with the current .elc format.

Even saving just the Lisp state as with “dumped.elc”, I think there could be state from the environment or build options that varies across distributions.  Lists of supported image types, distro customizations, things like that.  I’m not sure what benefit there is in trying to share saved Emacs state across distros.  If the goal is for a user to save a massively customized environment for future invocations, perhaps we should just work on speeding up the loading of the customizations.

If we want standardized object/executable format specifically for the preloaded environment, perhaps using the native format by way of the C compiler is a better choice.  I think this may have come up in the discussion before.  The big loss there is the ability to create a new saved environment without having a C compiler handy, but it seems like a thing few people are likely to want to do, and even fewer non-developers who might not be able to install a compiler.

Ken

Re: compiled lisp file format (Re: Skipping unexec via a big .elc file)

[Paul Eggert]

Ken Raeburn wrote:

> And I wouldn’t describe Guile’s “ELF everywhere” approach as entirely platform-neutral.

That's correct. It's more portable than what Emacs currently does, but it's less 
portable than saving state in .elc format.

> Even saving just the Lisp state as with “dumped.elc”, I think there could be state from the environment or build options that varies across distributions.

Yes, quite true. Even with "dumped .elc" or with any of the other methods 
proposed, it would be quite difficult to make the saved state portable to any 
platform. That kind of portability should not be our goal.

> If we want standardized object/executable format specifically for the preloaded environment, perhaps using the native format by way of the C compiler is a better choice.  I think this may have come up in the discussion before.  The big loss there is the ability to create a new saved environment without having a C compiler handy, but it seems like a thing few people are likely to want to do, and even fewer non-developers who might not be able to install a compiler.

Yes, this is my preferred solution too; I was the one who made that suggestion. 
Although Eli didn't like the idea at the time, perhaps there will come a day 
when we revisit it. It should be faster than even Guile's ELF-based loading, 
which is already plenty fast.

Re: compiled lisp file format (Re: Skipping unexec via a big .elc file)

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 29 May 2017 09:37:15 -0700
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> > If we want standardized object/executable format specifically for the preloaded environment, perhaps using the native format by way of the C compiler is a better choice.  I think this may have come up in the discussion before.  The big loss there is the ability to create a new saved environment without having a C compiler handy, but it seems like a thing few people are likely to want to do, and even fewer non-developers who might not be able to install a compiler.
> 
> Yes, this is my preferred solution too; I was the one who made that suggestion. 
> Although Eli didn't like the idea at the time, perhaps there will come a day 
> when we revisit it. It should be faster than even Guile's ELF-based loading, 
> which is already plenty fast.

I have no doubt it will be faster.  My problem with this alternative
is that I believe maintaining it will need experts on C and C
compilers that we cannot rely on having available.

Re: compiled lisp file format (Re: Skipping unexec via a big .elc file)

[Paul Eggert]

Eli Zaretskii wrote:
> My problem with this alternative
> is that I believe maintaining it will need experts on C and C
> compilers that we cannot rely on having available.

I don't see why we'd need any more expertise in C than we already require. The 
Emacs core is written in C, and one must be expert in C to maintain it. The C 
code that would be output would be simple -- much simpler than what we are 
already maintaining.

Re: compiled lisp file format (Re: Skipping unexec via a big .elc file)

[Eli Zaretskii]

> Cc: raeburn@raeburn.org, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 29 May 2017 11:03:38 -0700
> 
> Eli Zaretskii wrote:
> > My problem with this alternative
> > is that I believe maintaining it will need experts on C and C
> > compilers that we cannot rely on having available.
> 
> I don't see why we'd need any more expertise in C than we already require. The 
> Emacs core is written in C, and one must be expert in C to maintain it. The C 
> code that would be output would be simple -- much simpler than what we are 
> already maintaining.

This idea requires _generating_ C, something we don't currently do,
AFAIK.

As for whether it will be simple, I reserve my judgment, since no code
was presented to demonstrate the idea for some reasonably complex Lisp
data.

Re: compiled lisp file format (Re: Skipping unexec via a big .elc file)

[Paul Eggert]

Eli Zaretskii wrote:
> This idea requires_generating_  C, something we don't currently do

Actually, Emacs generates and reformats C code all the time, e.g., when editing 
and reindenting it. And the Emacs build procedure generates plenty of C code, 
e.g., lib/stdlib.h. It is not a stretch to assume enough C expertise to deal 
with this sort of thing.

> As for whether it will be simple, I reserve my judgment

That's discouraging.

Re: compiled lisp file format (Re: Skipping unexec via a big .elc file)

[Ken Raeburn]

Ah, yes, I remember this part of the discussion now… sorry, didn’t mean to stir up the same old argument again.

The expertise question would be an issue for attempting to adopt “ELF everywhere” in some fashion too, especially if we tried to do something interesting with using DWARF to store some kind of debug info.

On May 29, 2017, at 16:15, Paul Eggert <eggert@cs.ucla.edu> wrote:

> Eli Zaretskii wrote:
>> This idea requires_generating_  C, something we don't currently do
> 
> Actually, Emacs generates and reformats C code all the time, e.g., when editing and reindenting it. And the Emacs build procedure generates plenty of C code, e.g., lib/stdlib.h. It is not a stretch to assume enough C expertise to deal with this sort of thing.
> 
>> As for whether it will be simple, I reserve my judgment
> 
> That's discouraging.

I dunno, sounds like an invitation to produce an implementation that shows how straightforward it can be.  But, I had other things I was planning to work on tonight…

Ken

Re: compiled lisp file format (Re: Skipping unexec via a big .elc file)

[Eli Zaretskii]

> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 29 May 2017 13:15:28 -0700
> Cc: raeburn@raeburn.org, emacs-devel@gnu.org
> 
> Eli Zaretskii wrote:
> > This idea requires_generating_  C, something we don't currently do
> 
> Actually, Emacs generates and reformats C code all the time, e.g., when editing 
> and reindenting it. And the Emacs build procedure generates plenty of C code, 
> e.g., lib/stdlib.h. It is not a stretch to assume enough C expertise to deal 
> with this sort of thing.

I can only say I disagree.

> > As for whether it will be simple, I reserve my judgment
> 
> That's discouraging.

I'm sorry if you feel like that, but I don't see why: we are
discussing hypothetical code that I have no idea what it will look
like.  I just don't want to opine about something I never saw.  I
think it's reasonable.

Re: compiled lisp file format (Re: Skipping unexec via a big .elc file)

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 771 bytes --]

>>>>> "KR" == Ken Raeburn <raeburn@raeburn.org> writes:

KR> I haven’t had much time to further the work on the big-elc approach
KR> recently, but there is one idea I want to toss out there for possibly
KR> improving the load time further: Changing the .elc file format to a binary
KR> one. I’m not talking about a memory image like Daniel is working on. I
KR> mean a file representing a sequence of S-expressions, but optimized for
KR> loading speed rather than for human readability.

I would like to see this; I can't think of a reason not to encode the
information in the best format for loading.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 658 bytes --]

Re: Skipping unexec via a big .elc file

[Stefan Monnier]

> Perhaps we could have a separate, much smaller dumped.elc for batch
> invocations, to cater to these use cases.  Ken, does this make sense?

FWIW, I'm not sure if there is much to save there.


        Stefan

Re: Skipping unexec via a big .elc file

[Ken Raeburn]

On Apr 3, 2017, at 12:15, Ken Raeburn <raeburn@raeburn.org> wrote:

> On Mar 31, 2017, at 04:40, Ken Raeburn <raeburn@raeburn.org> wrote:
> 
>> 
>> On Mar 31, 2017, at 02:57, Eli Zaretskii <eliz@gnu.org> wrote:
>>> 
>>> This fixes the problem, and Emacs now starts OK, so the abbrevs issue
>>> is also solved.
>> 
>> Great!
>> 
>>> I think you should push all the changes you asked me to apply as
>>> patches.
>> 
>> Will do, probably this weekend.
> 
> Looks like the abbrev change isn’t actually working right… I got the quoting wrong, so the abbrev tables are constructed as (mostly) proper abbrev tables, and in the right order, but the “:parent” properties are bad. Working on fixing it up….

I’ve finally gotten what I think is a fixed version pushed to the branch, which should get the parent links right between abbrev tables.

It’s also got a fix to a problem I keep hitting doing parallel bootstrap builds.  (I goofed slightly on the log message though, got the name of the temp file wrong.)  I like to try to do bootstrap builds when testing and before pushing changes, so hopefully future fixes won’t be so tediously slow to check.

I’m still cleaning up my list of open issues, and will probably check that in in the branch’s admin/notes directory.

Ken

Re: Skipping unexec via a big .elc file

[Lars Brinkhoff]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
> FWIW, I just did a quick experiment with the patch below which dumps
> the state of Emacs's obarray after loadup.el into a big "dumped.elc"
> file.  [...]  So even if there might be ways to speed this up, it
> doesn't look too promising.

I suppose it's obvious that this dumped.elc can't easily be converted to
a c file which is compiled and linked into the final emacs.  For the
benefit of me and perhaps others that would otherwise waste time on
this, could someone just briefly explain why?

Re: Skipping unexec via a big .elc file

[Eli Zaretskii]

> From: Lars Brinkhoff <lars@nocrew.org>
> Date: Mon, 24 Oct 2016 20:34:49 +0200
> 
> I suppose it's obvious that this dumped.elc can't easily be converted to
> a c file which is compiled and linked into the final emacs.

It can.  More accurately, we could implement a back-end to the unexec
process that generates C source file.  That was Paul's suggestion.

I consider this option less desirable for several reasons:

  . writing and maintaining such a C back-end would be non-trivial,
    and would require good control of portable C programming,
    something that most of our contributors lack

  . it requires a C compiler, i.e. end-users cannot dump their own
    customized Emacs without having a compiler and linker installed

Re: When should ralloc.c be used?

[Stefan Monnier]

> I suppose you're right that we don't need to; we could instead hack on Emacs
> to get it to work without ralloc on recent glibc.

I thought it's just a matter of saying "don't use ralloc" (i.e. the use
of ralloc is only an optimization hack to try and avoid fragmentation
problems).


        Stefan

Re: When should ralloc.c be used?

[Stefan Monnier]

> I thought it's just a matter of saying "don't use ralloc" (i.e. the use
> of ralloc is only an optimization hack to try and avoid fragmentation
> problems).

And AFAICT we should just never use ralloc because the rest of Emacs's
code is actually not prepared to deal with the implications, and trying
to fix it is not only a lot of work, but would make the code less
maintainable.  I'd rather live with the fragmentation.


        Stefan

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 23 Oct 2016 10:28:22 -0400
> 
> I'd rather live with the fragmentation.

You mean, use gmalloc without ralloc?  Is that feasible?

Re: When should ralloc.c be used?

[Stefan Monnier]

>> I'd rather live with the fragmentation.
> You mean, use gmalloc without ralloc?

Not only when we use gmalloc but always.  I suggest we get rid of ralloc.c.

> Is that feasible?

Why not?


        Stefan

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 23 Oct 2016 11:07:23 -0400
> 
> >> I'd rather live with the fragmentation.
> > You mean, use gmalloc without ralloc?
> 
> Not only when we use gmalloc but always.  I suggest we get rid of ralloc.c.
> 
> > Is that feasible?
> 
> Why not?

I don't think we ever used such a configuration.  Is modern sbrk good
enough for gmalloc?

Re: When should ralloc.c be used?

[Stefan Monnier]

> I don't think we ever used such a configuration.  Is modern sbrk good
> enough for gmalloc?

Why not?


        Stefan

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 23 Oct 2016 12:30:32 -0400
> 
> > I don't think we ever used such a configuration.  Is modern sbrk good
> > enough for gmalloc?
> 
> Why not?

"Why not" is never a useful answer.

Re: When should ralloc.c be used?

[Stefan Monnier]

>> > I don't think we ever used such a configuration.  Is modern sbrk good
>> > enough for gmalloc?
>> Why not?
> "Why not" is never a useful answer.

It just means that I really see no reason why it wouldn't work just fine.
It's not like glibc's malloc was particularly magical, so we should be
able to do the same in gmalloc.c.


        Stefan

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 23 Oct 2016 12:49:29 -0400
> 
> >> > I don't think we ever used such a configuration.  Is modern sbrk good
> >> > enough for gmalloc?
> >> Why not?
> > "Why not" is never a useful answer.
> 
> It just means that I really see no reason why it wouldn't work just fine.
> It's not like glibc's malloc was particularly magical, so we should be
> able to do the same in gmalloc.c.

AAIK, glibc's malloc doesn't use sbrk anymore.

Re: When should ralloc.c be used?

[Stefan Monnier]

>> It just means that I really see no reason why it wouldn't work just fine.
>> It's not like glibc's malloc was particularly magical, so we should be
>> able to do the same in gmalloc.c.
> AAIK, glibc's malloc doesn't use sbrk anymore.

I don't think it matters very much since we use mmap for the buffers,
which is the main source of fragmentation otherwise, AFAIK.

And if it proves to really be a problem, we could replace our gmalloc.c
with a more recent one which builds on mmap.


        Stefan

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 23 Oct 2016 16:23:38 -0400
> 
> >> It just means that I really see no reason why it wouldn't work just fine.
> >> It's not like glibc's malloc was particularly magical, so we should be
> >> able to do the same in gmalloc.c.
> > AAIK, glibc's malloc doesn't use sbrk anymore.
> 
> I don't think it matters very much since we use mmap for the buffers,

No, we don't, not on GNU/Linux anyway.  Or do you see
USE_MMAP_FOR_BUFFERS defined to 1 in your src/config.h?

> And if it proves to really be a problem, we could replace our gmalloc.c
> with a more recent one which builds on mmap.

That's a possibility, yes.  But someone would have to bring such a
gmalloc, and probably leave the current one as well, to minimize the
impact on unaffected platforms.

Re: When should ralloc.c be used?

[Stefan Monnier]

>> I don't think it matters very much since we use mmap for the buffers,
> No, we don't, not on GNU/Linux anyway.

AFAIK the decision not to use mmap was due to the fact that glibc's
malloc itself uses mmap.  But if we don't use glibc's malloc, then why
wouldn't we decide to use mmap ourselves for the buffers?

> Or do you see USE_MMAP_FOR_BUFFERS defined to 1 in your src/config.h?

I'm not sure how to interpret what I see.  On Debian stable I see:

  Should Emacs use the GNU version of malloc?             yes
      (Using Doug Lea's new malloc from the GNU C Library.)
  Should Emacs use a relocating allocator for buffers?    no
  Should Emacs use mmap(2) for buffer allocation?         no

and on Debian testing I see:

  Should Emacs use the GNU version of malloc?             no (only before dumping)
  Should Emacs use a relocating allocator for buffers?    no
  Should Emacs use mmap(2) for buffer allocation?         no

so, in neither case do I see REL_ALLOC enabled.


        Stefan

Re: When should ralloc.c be used?

[Paul Eggert]

Stefan Monnier wrote:
> in neither case do I see REL_ALLOC enabled.

It looks like you are using the master branch. I think Eli is worried more 
urgently about the emacs-25 branch. For emacs-25 with bleeding-edge glibc, I 
would expect:

   Should Emacs use the GNU version of malloc?             yes
   Should Emacs use a relocating allocator for buffers?    yes
   Should Emacs use mmap(2) for buffer allocation?         no

because emacs-25 will compile both gmalloc.o and ralloc.o on such a platform.

Re: When should ralloc.c be used?

[Stefan Monnier]

>   Should Emacs use the GNU version of malloc?             yes
>   Should Emacs use a relocating allocator for buffers?    yes
>   Should Emacs use mmap(2) for buffer allocation?         no
> because emacs-25 will compile both gmalloc.o and ralloc.o on such a platform.

But I fail to see what's hard about changing that to "rel_alloc=no, mmap=yes".


        Stefan

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 24 Oct 2016 08:33:10 -0400
> Cc: emacs-devel@gnu.org
> 
> >   Should Emacs use the GNU version of malloc?             yes
> >   Should Emacs use a relocating allocator for buffers?    yes
> >   Should Emacs use mmap(2) for buffer allocation?         no
> > because emacs-25 will compile both gmalloc.o and ralloc.o on such a platform.
> 
> But I fail to see what's hard about changing that to "rel_alloc=no, mmap=yes".

Why do we need mmap at all?  Why not just use malloc (as implemented
by gmalloc)?

Using mmap has disadvantages: when you need to enlarge buffer text,
and that fails (because there are no more free pages/addresses after
the already allocated region), we need to copy buffer text to the new
allocation.  This happens quite a lot when we visit a compressed
buffer.  (The MS-Windows emulation of mmap in w32heap.c reserves twice
the number of pages as originally requested, for that very reason.)

So if we can stop using ralloc without also using mmap directly for
buffer text, that'd be a win, I think.

Re: When should ralloc.c be used?

[Stefan Monnier]

>> But I fail to see what's hard about changing that to "rel_alloc=no,
>> mmap=yes".
> Why do we need mmap at all?  Why not just use malloc (as implemented
> by gmalloc)?

AFAIU the reason we use ralloc is because of memory fragmentation, and
mmap brings similar benefits.  Maybe we don't need either of them, but
at least at some point in the past the fragmentation issue was
sufficient to convince people to write that code.


        Stefan

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 24 Oct 2016 10:12:27 -0400
> 
> >> But I fail to see what's hard about changing that to "rel_alloc=no,
> >> mmap=yes".
> > Why do we need mmap at all?  Why not just use malloc (as implemented
> > by gmalloc)?
> 
> AFAIU the reason we use ralloc is because of memory fragmentation, and
> mmap brings similar benefits.

But we have successfully used the glibc's malloc, without mmap, for
years without any sign of fragmentation problems.  So these
fragmentation problems are not as bad as they sound, at least in one
malloc implementation.

Re: When should ralloc.c be used?

[Stefan Monnier]

> But we have successfully used the glibc's malloc, without mmap, for
> years without any sign of fragmentation problems.

Yes, glibc's malloc was good enough.  I'm not sure that applies to
gmalloc.c.


        Stefan

Re: When should ralloc.c be used?

[Stefan Monnier]

> Using mmap has disadvantages: when you need to enlarge buffer text,
> and that fails (because there are no more free pages/addresses after
> the already allocated region), we need to copy buffer text to the new
> allocation.

All allocators suffer from this problem.  I haven't seen any evidence
that the mmap-based allocation code is significantly more prone to it.

Also, the glibc allocators used mmap internally when allocating
large-ish chunks (e.g. for buffer text), so if that was a problem, we
would have noticed, I think.

> (The MS-Windows emulation of mmap in w32heap.c reserves twice the
> number of pages as originally requested, for that very reason.)

Indeed, if this problem proves significant, there are fairly easy ways
to reduce its impact, such as using the kind of approach you mention.

Another advantage of using mmap is that it can return the memory to the
OS once you kill your large buffer, whereas with gmalloc+ralloc this
basically never happens, AFAIK.


        Stefan

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: eggert@cs.ucla.edu,  emacs-devel@gnu.org
> Date: Mon, 24 Oct 2016 10:37:19 -0400
> 
> > Using mmap has disadvantages: when you need to enlarge buffer text,
> > and that fails (because there are no more free pages/addresses after
> > the already allocated region), we need to copy buffer text to the new
> > allocation.
> 
> All allocators suffer from this problem.  I haven't seen any evidence
> that the mmap-based allocation code is significantly more prone to it.

I have seen that.  The native glibc malloc, the on GNU/Linux systems
were using until we got screwed by the recent glibc, didn't have this
problem, while mmap-based allocator did.  Don't ask me how glibc does
it, I don't know; but the fact is there.  This was discovered when the
Windows mmap emulation in w32heap.c was developed and tested.

> Also, the glibc allocators used mmap internally when allocating
> large-ish chunks (e.g. for buffer text), so if that was a problem, we
> would have noticed, I think.

True; but they somehow work around the problem.

> Another advantage of using mmap is that it can return the memory to the
> OS once you kill your large buffer, whereas with gmalloc+ralloc this
> basically never happens, AFAIK.

Not entirely true: ralloc calls the system sbrk with a negative
argument when it feels like it.

Re: When should ralloc.c be used?

[Daniel Colascione]

On 10/24/2016 08:40 AM, Eli Zaretskii wrote:
>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: eggert@cs.ucla.edu,  emacs-devel@gnu.org
>> Date: Mon, 24 Oct 2016 10:37:19 -0400
>>
>>> Using mmap has disadvantages: when you need to enlarge buffer text,
>>> and that fails (because there are no more free pages/addresses after
>>> the already allocated region), we need to copy buffer text to the new
>>> allocation.

64-bit address spaces are *huge*. What about just making every buffer 
allocation 2GB long or so, marked PROT_NONE? You don't actually have to 
commit all that memory --- all you've done is set aside that address 
space. But because you've set aside so much address space, you'll very 
likely be able to expand the actual allocation region (a subset of the 
reserved region) as much as you want.

>> All allocators suffer from this problem.  I haven't seen any evidence
>> that the mmap-based allocation code is significantly more prone to it.
>
> I have seen that.  The native glibc malloc, the on GNU/Linux systems
> were using until we got screwed by the recent glibc, didn't have this
> problem, while mmap-based allocator did.  Don't ask me how glibc does
> it, I don't know; but the fact is there.  This was discovered when the
> Windows mmap emulation in w32heap.c was developed and tested.
>
>> Also, the glibc allocators used mmap internally when allocating
>> large-ish chunks (e.g. for buffer text), so if that was a problem, we
>> would have noticed, I think.
>
> True; but they somehow work around the problem.
>
>> Another advantage of using mmap is that it can return the memory to the
>> OS once you kill your large buffer, whereas with gmalloc+ralloc this
>> basically never happens, AFAIK.
>
> Not entirely true: ralloc calls the system sbrk with a negative
> argument when it feels like it.

You can also madvise(MADV_DONTNEED, ...) regions *inside* the heap that 
contain only freed memory. This procedure also returns memory to the 
operating system.

Re: When should ralloc.c be used?

[Eli Zaretskii]

> Cc: eggert@cs.ucla.edu, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Mon, 24 Oct 2016 09:27:43 -0700
> 
> >>> Using mmap has disadvantages: when you need to enlarge buffer text,
> >>> and that fails (because there are no more free pages/addresses after
> >>> the already allocated region), we need to copy buffer text to the new
> >>> allocation.
> 
> 64-bit address spaces are *huge*. What about just making every buffer 
> allocation 2GB long or so, marked PROT_NONE? You don't actually have to 
> commit all that memory --- all you've done is set aside that address 
> space. But because you've set aside so much address space, you'll very 
> likely be able to expand the actual allocation region (a subset of the 
> reserved region) as much as you want.

Sounds OK, although I'm not an expert on that.  But in any case, these
ideas are not baked enough to be applied to the release branch, if we
want to release Emacs 25.2 soon (as in "in a couple of months").

(Of course, there's always the case of a file larger than 2GB, it's
not unheard of, although still quite rare.)

Re: When should ralloc.c be used?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > 64-bit address spaces are *huge*. What about just making every buffer 
  > allocation 2GB long or so, marked PROT_NONE?

Does Linux handles such sparseness efficiently?
I don't know.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used?

[Stefan Monnier]

> 64-bit address spaces are *huge*. What about just making every buffer
> allocation 2GB long or so, marked PROT_NONE?

Won't be sufficient for 3GB buffers, obviously ;-)


        Stefan

Re: When should ralloc.c be used?

[Stefan Monnier]

>> Not entirely true: ralloc calls the system sbrk with a negative
>> argument when it feels like it.
> You can also madvise(MADV_DONTNEED, ...) regions *inside* the heap that
> contain only freed memory. This procedure also returns memory to the
> operating system.

ralloc.c doesn't do this currently.


        Stefan

Re: When should ralloc.c be used?

[Jérémie Courrèges-Anglas]

Daniel Colascione <dancol@dancol.org> writes:

> On 10/24/2016 08:40 AM, Eli Zaretskii wrote:
>>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>>> Cc: eggert@cs.ucla.edu,  emacs-devel@gnu.org
>>> Date: Mon, 24 Oct 2016 10:37:19 -0400
>>>
>>>> Using mmap has disadvantages: when you need to enlarge buffer text,
>>>> and that fails (because there are no more free pages/addresses after
>>>> the already allocated region), we need to copy buffer text to the new
>>>> allocation.
>
> 64-bit address spaces are *huge*. What about just making every buffer
> allocation 2GB long or so, marked PROT_NONE? You don't actually have to
> commit all that memory --- all you've done is set aside that address
> space.

IIUC you suggest relying on memory overcommit.  That doesn't sound
portable at all.  Not all OSes do overcommit and the ones who do
generally provide a way to disable it.

-- 
jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF  DDCC 0DFA 74AE 1524 E7EE

Re: When should ralloc.c be used?

[Daniel Colascione]

jca@wxcvbn.org (Jérémie Courrèges-Anglas) writes:

> Daniel Colascione <dancol@dancol.org> writes:
>
>> On 10/24/2016 08:40 AM, Eli Zaretskii wrote:
>>>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>>>> Cc: eggert@cs.ucla.edu,  emacs-devel@gnu.org
>>>> Date: Mon, 24 Oct 2016 10:37:19 -0400
>>>>
>>>>> Using mmap has disadvantages: when you need to enlarge buffer text,
>>>>> and that fails (because there are no more free pages/addresses after
>>>>> the already allocated region), we need to copy buffer text to the new
>>>>> allocation.
>>
>> 64-bit address spaces are *huge*. What about just making every buffer
>> allocation 2GB long or so, marked PROT_NONE? You don't actually have to
>> commit all that memory --- all you've done is set aside that address
>> space.
>
> IIUC you suggest relying on memory overcommit.  That doesn't sound
> portable at all.  Not all OSes do overcommit and the ones who do
> generally provide a way to disable it.

You understand incorrectly. "Overcommit" is the practice of allowing an
operating system to lie about how much memory it's guaranteed to give
applications in the future.  We're not talking about guaranteed
memory. We're talking about setting aside address space only, not asking
the OS to make guarantees about future memory availability.  All major
operating systems, even ones like Windows that don't do overcommit,
provide ways to reserve address space without asking the OS to guarantee
availability of memory.

That said, my idea probably isn't the best --- but it doesn't rely
on overcommit.

Re: When should ralloc.c be used?

[Jérémie Courrèges-Anglas]

Daniel Colascione <dancol@dancol.org> writes:

> jca@wxcvbn.org (Jérémie Courrèges-Anglas) writes:
>
>> Daniel Colascione <dancol@dancol.org> writes:
>>
>>> On 10/24/2016 08:40 AM, Eli Zaretskii wrote:
>>>>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>>>>> Cc: eggert@cs.ucla.edu,  emacs-devel@gnu.org
>>>>> Date: Mon, 24 Oct 2016 10:37:19 -0400
>>>>>
>>>>>> Using mmap has disadvantages: when you need to enlarge buffer text,
>>>>>> and that fails (because there are no more free pages/addresses after
>>>>>> the already allocated region), we need to copy buffer text to the new
>>>>>> allocation.
>>>
>>> 64-bit address spaces are *huge*. What about just making every buffer
>>> allocation 2GB long or so, marked PROT_NONE? You don't actually have to
>>> commit all that memory --- all you've done is set aside that address
>>> space.
>>
>> IIUC you suggest relying on memory overcommit.  That doesn't sound
>> portable at all.  Not all OSes do overcommit and the ones who do
>> generally provide a way to disable it.
>
> You understand incorrectly. "Overcommit" is the practice of allowing an
> operating system to lie about how much memory it's guaranteed to give
> applications in the future.  We're not talking about guaranteed
> memory. We're talking about setting aside address space only, not asking
> the OS to make guarantees about future memory availability.  All major
> operating systems, even ones like Windows that don't do overcommit,
> provide ways to reserve address space without asking the OS to guarantee
> availability of memory.

Can you point at some documentation regarding those techniques?  I fail
to find one that would work on my "non-major", mostly POSIX OS.

> That said, my idea probably isn't the best --- but it doesn't rely
> on overcommit.

-- 
jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF  DDCC 0DFA 74AE 1524 E7EE

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Daniel Colascione <dancol@dancol.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  eggert@cs.ucla.edu,  emacs-devel@gnu.org
> Date: Thu, 27 Oct 2016 23:23:05 -0700
> 
> We're talking about setting aside address space only, not asking
> the OS to make guarantees about future memory availability.  All major
> operating systems, even ones like Windows that don't do overcommit,
> provide ways to reserve address space without asking the OS to guarantee
> availability of memory.

Not sure I understand you: if a portion of the address space has been
reserved, how come these addresses won't be available when we try to
commit them later?  There might not be physical pages available for
that, but virtual memory for those addresses must be available, no?

Re: When should ralloc.c be used?

[Daniel Colascione]

On 10/28/2016 12:46 AM, Eli Zaretskii wrote:
>> From: Daniel Colascione <dancol@dancol.org>
>> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  eggert@cs.ucla.edu,  emacs-devel@gnu.org
>> Date: Thu, 27 Oct 2016 23:23:05 -0700
>>
>> We're talking about setting aside address space only, not asking
>> the OS to make guarantees about future memory availability.  All major
>> operating systems, even ones like Windows that don't do overcommit,
>> provide ways to reserve address space without asking the OS to guarantee
>> availability of memory.
>
> Not sure I understand you: if a portion of the address space has been
> reserved, how come these addresses won't be available when we try to
> commit them later?  There might not be physical pages available for
> that, but virtual memory for those addresses must be available, no?

I'm not sure I understand what you're confused about, so I'll try a 
broader explanation.

Say I mmap (anonymously, for simplicity) a page PROT_NONE. After the 
initial mapping, that address space is unavailable for other uses. But 
because the page protections are PROT_NONE, my program has no legal 
right to access that page, so the OS doesn't have to guarantee that it 
can find a physical page to back that page I've mmaped. In this state, 
the memory is reserved.

The 20GB PROT_NONE address space reservation itself requires very little 
memory. It's just a note in the kernel's VM interval tree that says "the 
addresses in range [0x20000, 0x500020000) are reserved". Virtual memory is

Now imagine I change the protections to PROT_READ|PROT_WRITE --- once 
the PROT_READ|PROT_WRITE mprotect succeeds, my program has every right 
to access that page; under a strict accounting scheme (that is, without 
overcommit), the OS has to guarantee that it'll be able to go find a 
physical page to back that virtual page. In this state, the memory is 
committed -- the kernel has committed to finding backing storage for 
that page at some point when the current process tries to access it.

Say you have a strict-accounting system with 1GB of RAM and 1GB of swap. 
I can write a program that reserves 20GB of address space. That's fine. 
The kernel isn't promising to give you 20GB of memory: it's setting 
address space. Now if I attempt to map 20GB PROT_READ|PROT_WRITE, on any 
reasonable (i..e, not overcommit) system, mmap should fail, since 
there's no way a system with 1GB of RAM and 1GB of swap can promise to 
provide 20GB of private memory.

Overcommit confuses the issue: the kernel will _commit_ to as much 
memory as you ask it for and then renege on that commitment when it 
finds it convenient. An overcommit system with 1GB of RAM and 1GB of 
swap will happily let you make that 20GB PROT_READ|PROT_WRITE mapping. 
It'll just kill you after you use more than 2GB of that mapping. A 
non-overcommit system understands how to say "sorry, I can't let you do 
that" up front. On a non-overcommit system, your process will never be 
killed for accessing memory that the kernel told the process in advance 
that it could use.

I think Jérémie is working with a mental model where every memory 
mapping is commit, and only an overcommit system allows you to commit 
more memory than the system actually has. Any system will let you 
reserve more address space than you have available commit: reservations 
are cheap.

(In Windows, the corresponding concepts MEM_RESERVE and MEM_COMMIT. 
Windows is much more explicit about the difference between memory 
reservations and memory commitments than Linux is, because most of the 
time, Linux users don't care about the difference.)

Re: When should ralloc.c be used?

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, eggert@cs.ucla.edu, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Fri, 28 Oct 2016 01:11:08 -0700
> 
> Say I mmap (anonymously, for simplicity) a page PROT_NONE. After the 
> initial mapping, that address space is unavailable for other uses. But 
> because the page protections are PROT_NONE, my program has no legal 
> right to access that page, so the OS doesn't have to guarantee that it 
> can find a physical page to back that page I've mmaped. In this state, 
> the memory is reserved.
> 
> The 20GB PROT_NONE address space reservation itself requires very little 
> memory. It's just a note in the kernel's VM interval tree that says "the 
> addresses in range [0x20000, 0x500020000) are reserved". Virtual memory is
> 
> Now imagine I change the protections to PROT_READ|PROT_WRITE --- once 
> the PROT_READ|PROT_WRITE mprotect succeeds, my program has every right 
> to access that page; under a strict accounting scheme (that is, without 
> overcommit), the OS has to guarantee that it'll be able to go find a 
> physical page to back that virtual page. In this state, the memory is 
> committed -- the kernel has committed to finding backing storage for 
> that page at some point when the current process tries to access it.

I'm with you up to here.  My question is whether PROT_READ|PROT_WRITE
call could fail after PROT_NONE succeeded.  You seem to say it could;
I thought it couldn't.

> Say you have a strict-accounting system with 1GB of RAM and 1GB of swap. 
> I can write a program that reserves 20GB of address space.

I thought such a reservation should fail, because you don't have
enough virtual memory for 20GB of addresses.  IOW, I thought the
ability to reserve address space is restricted by the actual amount of
virtual memory available on the system at the time of the call.  You
seem to say I was wrong.

Re: When should ralloc.c be used?

[Daniel Colascione]

On 10/28/2016 01:27 AM, Eli Zaretskii wrote:
>> Cc: monnier@iro.umontreal.ca, eggert@cs.ucla.edu, emacs-devel@gnu.org
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Fri, 28 Oct 2016 01:11:08 -0700
>>
>> Say I mmap (anonymously, for simplicity) a page PROT_NONE. After the
>> initial mapping, that address space is unavailable for other uses. But
>> because the page protections are PROT_NONE, my program has no legal
>> right to access that page, so the OS doesn't have to guarantee that it
>> can find a physical page to back that page I've mmaped. In this state,
>> the memory is reserved.
>>
>> The 20GB PROT_NONE address space reservation itself requires very little
>> memory. It's just a note in the kernel's VM interval tree that says "the
>> addresses in range [0x20000, 0x500020000) are reserved". Virtual memory is
>>
>> Now imagine I change the protections to PROT_READ|PROT_WRITE --- once
>> the PROT_READ|PROT_WRITE mprotect succeeds, my program has every right
>> to access that page; under a strict accounting scheme (that is, without
>> overcommit), the OS has to guarantee that it'll be able to go find a
>> physical page to back that virtual page. In this state, the memory is
>> committed -- the kernel has committed to finding backing storage for
>> that page at some point when the current process tries to access it.
>
> I'm with you up to here.  My question is whether PROT_READ|PROT_WRITE
> call could fail after PROT_NONE succeeded.  You seem to say it could;
> I thought it couldn't.

Yes, it can fail. This program just failed on my system, which is a 
strict accounting (echo 2 > /proc/sys/vm/overcommit_memory) Linux box 
with much less than 100GB total commit available.

#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <errno.h>

size_t GB = (size_t) 1024 * 1024 * 1024;
int
main()
{
     size_t sz = 100*GB;
     void* mem = mmap(NULL, sz, PROT_NONE,
                      MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
     if (mem == MAP_FAILED) {
         fprintf(stderr, "map failed: %s\n", strerror(errno));
         return 1;
     }

     if (mprotect(mem, sz, PROT_READ|PROT_WRITE)) {
         fprintf(stderr, "mprotect failed: %s\n", strerror(errno));
         return 1;
     }

     fprintf(stderr, "mprotect worked\n");
     return 0;
}

>> Say you have a strict-accounting system with 1GB of RAM and 1GB of swap.
>> I can write a program that reserves 20GB of address space.
>
> I thought such a reservation should fail, because you don't have
> enough virtual memory for 20GB of addresses.  IOW, I thought the
> ability to reserve address space is restricted by the actual amount of
> virtual memory available on the system at the time of the call.  You
> seem to say I was wrong.

I'm not sure you're even wrong :-) What does "virtual memory" mean to 
you? I'm not sure what you have in mind maps to any of the concepts I'm 
using.

When we allocate memory, we can consume two resources: address space and 
commit. That 100GB mmap above doesn't consume virtual memory, but it 
does consume address space. Address space is a finite resource, but 
usually much larger than commit, which is the sum of RAM and swap space. 
When you commit a page, the resource you're consuming is commit.

(Technically, the 100GB mapping consumes real memory enough for the OS 
to remember you've set aside that address space, but it's usually a 
negligible book-keeping note. On my system, I can make sz equal to 80TB 
or so before the mmap starts to fail: that's about the size of the 
address space range dictated by amd64 processor design.)

(In a 32-bit process on modern systems, it's frequently the case that 
you have more commit on the system than any one process has address space.)

Re: When should ralloc.c be used?

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, eggert@cs.ucla.edu, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Fri, 28 Oct 2016 01:44:33 -0700
> 
> >> Say you have a strict-accounting system with 1GB of RAM and 1GB of swap.
> >> I can write a program that reserves 20GB of address space.
> >
> > I thought such a reservation should fail, because you don't have
> > enough virtual memory for 20GB of addresses.  IOW, I thought the
> > ability to reserve address space is restricted by the actual amount of
> > virtual memory available on the system at the time of the call.  You
> > seem to say I was wrong.
> 
> I'm not sure you're even wrong :-) What does "virtual memory" mean to 
> you?

Physical + swap, as usual.

> When we allocate memory, we can consume two resources: address space and 
> commit. That 100GB mmap above doesn't consume virtual memory, but it 
> does consume address space. Address space is a finite resource, but 
> usually much larger than commit, which is the sum of RAM and swap space. 
> When you commit a page, the resource you're consuming is commit.

If reserving a range of addresses doesn't necessarily mean they will
be later available for committing, then what is the purpose of
reserving them in the first place?  What good does it do?

We have in w32heap.c:mmap_realloc code that attempts to commit pages
that were previously reserved.  That code does recover from a failure
to commit, but such a failure is deemed unusual and causes special
warnings under debugger.  I never saw these warnings happen, except
when we had bugs in that code.  You seem to say that this is based on
false premises, and there's nothing unusual about MEM_COMMIT to fail
for the range of pages previously reserved with MEM_RESERVE.

Re: When should ralloc.c be used?

[Daniel Colascione]

On 10/28/2016 02:43 AM, Eli Zaretskii wrote:
>> Cc: monnier@iro.umontreal.ca, eggert@cs.ucla.edu, emacs-devel@gnu.org
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Fri, 28 Oct 2016 01:44:33 -0700
>>
>>>> Say you have a strict-accounting system with 1GB of RAM and 1GB of swap.
>>>> I can write a program that reserves 20GB of address space.
>>>
>>> I thought such a reservation should fail, because you don't have
>>> enough virtual memory for 20GB of addresses.  IOW, I thought the
>>> ability to reserve address space is restricted by the actual amount of
>>> virtual memory available on the system at the time of the call.  You
>>> seem to say I was wrong.
>>
>> I'm not sure you're even wrong :-) What does "virtual memory" mean to
>> you?
>
> Physical + swap, as usual.
>
>> When we allocate memory, we can consume two resources: address space and
>> commit. That 100GB mmap above doesn't consume virtual memory, but it
>> does consume address space. Address space is a finite resource, but
>> usually much larger than commit, which is the sum of RAM and swap space.
>> When you commit a page, the resource you're consuming is commit.
>
> If reserving a range of addresses doesn't necessarily mean they will
> be later available for committing, then what is the purpose of
> reserving them in the first place?  What good does it do?

Reserving address space is useful for making sure you have a contiguous 
range of virtual addresses that you can use later.

> We have in w32heap.c:mmap_realloc code that attempts to commit pages
> that were previously reserved.  That code does recover from a failure
> to commit, but such a failure is deemed unusual and causes special
> warnings under debugger.  I never saw these warnings happen, except
> when we had bugs in that code.  You seem to say that this is based on
> false premises, and there's nothing unusual about MEM_COMMIT to fail
> for the range of pages previously reserved with MEM_RESERVE.

The MEM_COMMIT failure might be rare in practice --- systems have a lot 
of memory these days --- but MEM_COMMIT failing for a memory region 
previously reserved with MEM_RESERVE is perfectly legal. MEM_RESERVE 
does not stake a claim on the system's memory resources. It consumes 
only your own address space.

Re: When should ralloc.c be used?

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, eggert@cs.ucla.edu, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Fri, 28 Oct 2016 02:52:19 -0700
> 
> > If reserving a range of addresses doesn't necessarily mean they will
> > be later available for committing, then what is the purpose of
> > reserving them in the first place?  What good does it do?
> 
> Reserving address space is useful for making sure you have a contiguous 
> range of virtual addresses that you can use later.

But if committing more pages from the reserved range is not guaranteed
to succeed, I cannot rely on getting that contiguous range of
addresses, can I?

> > We have in w32heap.c:mmap_realloc code that attempts to commit pages
> > that were previously reserved.  That code does recover from a failure
> > to commit, but such a failure is deemed unusual and causes special
> > warnings under debugger.  I never saw these warnings happen, except
> > when we had bugs in that code.  You seem to say that this is based on
> > false premises, and there's nothing unusual about MEM_COMMIT to fail
> > for the range of pages previously reserved with MEM_RESERVE.
> 
> The MEM_COMMIT failure might be rare in practice --- systems have a lot 
> of memory these days --- but MEM_COMMIT failing for a memory region 
> previously reserved with MEM_RESERVE is perfectly legal.

I can only say that I never saw that happening.

Thanks.

Re: When should ralloc.c be used?

[Stefan Monnier]

> But if committing more pages from the reserved range is not guaranteed
> to succeed, I cannot rely on getting that contiguous range of
> addresses, can I?

It should only fail in those cases where a new mmap (or malloc) would
also fail.


        Stefan

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 28 Oct 2016 09:37:04 -0400
> 
> > But if committing more pages from the reserved range is not guaranteed
> > to succeed, I cannot rely on getting that contiguous range of
> > addresses, can I?
> 
> It should only fail in those cases where a new mmap (or malloc) would
> also fail.

That means never, for all practical purposes.

Re: When should ralloc.c be used?

[Stefan Monnier]

>> > But if committing more pages from the reserved range is not guaranteed
>> > to succeed, I cannot rely on getting that contiguous range of
>> > addresses, can I?
>> It should only fail in those cases where a new mmap (or malloc) would
>> also fail.
> That means never, for all practical purposes.

Of course,


        Stefan

Re: When should ralloc.c be used?

[Daniel Colascione]

On 10/28/2016 05:25 AM, Eli Zaretskii wrote:
>> Cc: monnier@iro.umontreal.ca, eggert@cs.ucla.edu, emacs-devel@gnu.org
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Fri, 28 Oct 2016 02:52:19 -0700
>>
>>> If reserving a range of addresses doesn't necessarily mean they will
>>> be later available for committing, then what is the purpose of
>>> reserving them in the first place?  What good does it do?
>>
>> Reserving address space is useful for making sure you have a contiguous
>> range of virtual addresses that you can use later.
>
> But if committing more pages from the reserved range is not guaranteed
> to succeed, I cannot rely on getting that contiguous range of
> addresses, can I?

You already _have_ the range of addresses. You just can't do anything 
with them yet.

Here's another use case: magic ring buffers. (Where you put two 
consecutive views of the same file in memory next to each other so that 
operations on the ring buffer don't need to be split even in cases where 
they'd wrap the end of the ring.)

Say on our 1GB RAM, 1GB swap system we want to memory-map a 5GB ring 
buffer log file. We can do it safely and atomically like this:

1) Reserve 10GB of address space with an anonymous PROT_NONE mapping; 
the mapping is at $ADDR
2) Memory-map our log file at $ADDR with PROT_READ|PROT_WRITE; (the 
mapping is file-backed, not anonymous, so it doesn't count against 
system commit charge)
3) Memory-map the log file _again_ at $ADDR+5GB

Now we have a nice mirrored view of our ring buffer, and thanks to the 
PROT_NONE mapping we set up in step one, no other thread was able to 
sneak in the middle and allocate something in the [$ADDR+5GB,$ADDR+10GB) 
range and spoil our ability to set up the mirroring.

In this instance, setting aside address space without allocating backing 
storage for it turned out to be very useful.

>
>>> We have in w32heap.c:mmap_realloc code that attempts to commit pages
>>> that were previously reserved.  That code does recover from a failure
>>> to commit, but such a failure is deemed unusual and causes special
>>> warnings under debugger.  I never saw these warnings happen, except
>>> when we had bugs in that code.  You seem to say that this is based on
>>> false premises, and there's nothing unusual about MEM_COMMIT to fail
>>> for the range of pages previously reserved with MEM_RESERVE.
>>
>> The MEM_COMMIT failure might be rare in practice --- systems have a lot
>> of memory these days --- but MEM_COMMIT failing for a memory region
>> previously reserved with MEM_RESERVE is perfectly legal.
>
> I can only say that I never saw that happening.
>
> Thanks.
>

Re: When should ralloc.c be used?

[Eli Zaretskii]

> Cc: monnier@iro.umontreal.ca, eggert@cs.ucla.edu, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Fri, 28 Oct 2016 08:41:48 -0700
> 
> >> Reserving address space is useful for making sure you have a contiguous
> >> range of virtual addresses that you can use later.
> >
> > But if committing more pages from the reserved range is not guaranteed
> > to succeed, I cannot rely on getting that contiguous range of
> > addresses, can I?
> 
> You already _have_ the range of addresses. You just can't do anything 
> with them yet.

It's no use "having" the addresses, in the above sense, if I can't
rely on being able to do anything with them later.

> Here's another use case: magic ring buffers. (Where you put two 
> consecutive views of the same file in memory next to each other so that 
> operations on the ring buffer don't need to be split even in cases where 
> they'd wrap the end of the ring.)
> 
> Say on our 1GB RAM, 1GB swap system we want to memory-map a 5GB ring 
> buffer log file. We can do it safely and atomically like this:
> 
> 1) Reserve 10GB of address space with an anonymous PROT_NONE mapping; 
> the mapping is at $ADDR
> 2) Memory-map our log file at $ADDR with PROT_READ|PROT_WRITE; (the 
> mapping is file-backed, not anonymous, so it doesn't count against 
> system commit charge)
> 3) Memory-map the log file _again_ at $ADDR+5GB

If 3) fails, what do you do?

> Now we have a nice mirrored view of our ring buffer, and thanks to the 
> PROT_NONE mapping we set up in step one, no other thread was able to 
> sneak in the middle and allocate something in the [$ADDR+5GB,$ADDR+10GB) 
> range and spoil our ability to set up the mirroring.
> 
> In this instance, setting aside address space without allocating backing 
> storage for it turned out to be very useful.

Not if PROT_READ|PROT_WRITE call fails.

But if, as Stefan says, this will "never" happen, then the problem
doesn't exist in practice, and for all practical purposes what I
thought should happen, does happen, even if in theory it can fail.

Re: When should ralloc.c be used?

[Daniel Colascione]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: monnier@iro.umontreal.ca, eggert@cs.ucla.edu, emacs-devel@gnu.org
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Fri, 28 Oct 2016 08:41:48 -0700
>> 
>> >> Reserving address space is useful for making sure you have a contiguous
>> >> range of virtual addresses that you can use later.
>> >
>> > But if committing more pages from the reserved range is not guaranteed
>> > to succeed, I cannot rely on getting that contiguous range of
>> > addresses, can I?
>> 
>> You already _have_ the range of addresses. You just can't do anything 
>> with them yet.
>
> It's no use "having" the addresses, in the above sense, if I can't
> rely on being able to do anything with them later.

You can rely on nobody else using that address space, though. This
exclusion in itself is valuable. It's like an electric company buying
right-of-way for a high-voltage transmission line.  Sure, the electric
company isn't doing anything with that long strip of land, but the value
is in nobody _else_ doing anything with it either.

>
>> Here's another use case: magic ring buffers. (Where you put two 
>> consecutive views of the same file in memory next to each other so that 
>> operations on the ring buffer don't need to be split even in cases where 
>> they'd wrap the end of the ring.)
>> 
>> Say on our 1GB RAM, 1GB swap system we want to memory-map a 5GB ring 
>> buffer log file. We can do it safely and atomically like this:
>> 
>> 1) Reserve 10GB of address space with an anonymous PROT_NONE mapping; 
>> the mapping is at $ADDR
>> 2) Memory-map our log file at $ADDR with PROT_READ|PROT_WRITE; (the 
>> mapping is file-backed, not anonymous, so it doesn't count against 
>> system commit charge)
>> 3) Memory-map the log file _again_ at $ADDR+5GB
>
> If 3) fails, what do you do?

Unmap the original mapping and mapping #2, then fail the higher-level
make_magic_ring_buffer operation. Any operation that allocates memory
can fail.

>
>> Now we have a nice mirrored view of our ring buffer, and thanks to the 
>> PROT_NONE mapping we set up in step one, no other thread was able to 
>> sneak in the middle and allocate something in the [$ADDR+5GB,$ADDR+10GB) 
>> range and spoil our ability to set up the mirroring.
>> 
>> In this instance, setting aside address space without allocating backing 
>> storage for it turned out to be very useful.
>
> Not if PROT_READ|PROT_WRITE call fails.
>
> But if, as Stefan says, this will "never" happen, then the problem
> doesn't exist in practice, and for all practical purposes what I
> thought should happen, does happen, even if in theory it can fail.

It's unlikely, but it's a legal failure mode.

Re: When should ralloc.c be used?

[Stefan Monnier]

> If reserving a range of addresses doesn't necessarily mean they will
> be later available for committing, then what is the purpose of
> reserving them in the first place?  What good does it do?

My guess is that you can later use that address space to mmap files
in there.  Equivalently, you could increase the swap space between the
time you PROT_NONE and the time you switch to PROT_RW.

PROT_NONE is useful in a situation such as ours: you want to mmap
a hundred buffers, and make sure you can grow any of them without
knowing beforehand which one will grow.

But most likely, whether it's useful or not to be able to reserve 80TB
of address space even if you'd never be able to PROT_RW later was not
really relevant to the design of the API.


        Stefan

Re: When should ralloc.c be used?

[Jérémie Courrèges-Anglas]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: monnier@iro.umontreal.ca, eggert@cs.ucla.edu, emacs-devel@gnu.org
>> From: Daniel Colascione <dancol@dancol.org>
>> Date: Fri, 28 Oct 2016 01:11:08 -0700
>> 
>> Say I mmap (anonymously, for simplicity) a page PROT_NONE. After the 
>> initial mapping, that address space is unavailable for other uses. But 
>> because the page protections are PROT_NONE, my program has no legal 
>> right to access that page, so the OS doesn't have to guarantee that it 
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> can find a physical page to back that page I've mmaped. In this state, 
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This is what I think is a problem in your reasoning.  "Doesn't have to
guarantee" doesn't mean that the kernel *should not* actually check the
available memory and resource limits.

>> the memory is reserved.
>> 
>> The 20GB PROT_NONE address space reservation itself requires very little 
>> memory. It's just a note in the kernel's VM interval tree that says "the 
>> addresses in range [0x20000, 0x500020000) are reserved". Virtual memory is
>> 
>> Now imagine I change the protections to PROT_READ|PROT_WRITE --- once 
>> the PROT_READ|PROT_WRITE mprotect succeeds, my program has every right 
>> to access that page; under a strict accounting scheme (that is, without 
>> overcommit), the OS has to guarantee that it'll be able to go find a 
>> physical page to back that virtual page. In this state, the memory is 
>> committed -- the kernel has committed to finding backing storage for 
>> that page at some point when the current process tries to access it.
>
> I'm with you up to here.  My question is whether PROT_READ|PROT_WRITE
> call could fail after PROT_NONE succeeded.  You seem to say it could;
> I thought it couldn't.

I wouldn't have thought that PROT_NONE vs PROT_READ|PROT_WRITE would
have changed anything here, but on *some* OSes it does, however it is
not portable.  At least OpenBSD doesn't behave like what you describe.
IMHO people who rely on this kind of reservations rely on
implementation-defined behavior.

Also, sanity wise, I'd prefer having mmap(2) fail right away rather than
having mprotect(2) fail, much later.  *If* mprotect(2) actually fails ;
of course, you don't want to play russian roulette with your OS's
flavor of the OOM-killer either.

>> Say you have a strict-accounting system with 1GB of RAM and 1GB of swap. 
>> I can write a program that reserves 20GB of address space.
>
> I thought such a reservation should fail, because you don't have
> enough virtual memory for 20GB of addresses.  IOW, I thought the
> ability to reserve address space is restricted by the actual amount of
> virtual memory available on the system at the time of the call.  You
> seem to say I was wrong.


-- 
jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF  DDCC 0DFA 74AE 1524 E7EE

Re: When should ralloc.c be used?

[Stefan Monnier]

> I wouldn't have thought that PROT_NONE vs PROT_READ|PROT_WRITE would
> have changed anything here, but on *some* OSes it does, however it is
> not portable.  At least OpenBSD doesn't behave like what you describe.

Are you sure?  Can you point to concrete evidence?

Not that's it's important (using a hard-coded number like 2GB wouldn't
work, so we'd more likely use something like w32heap.c's "pre-allocate
double the size", which doesn't suffer from that problem anyway and
still guarantees efficient behavior when growing a buffer progressively
from 1B to 100GB).


        Stefan

Re: When should ralloc.c be used?

[Jérémie Courrèges-Anglas]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I wouldn't have thought that PROT_NONE vs PROT_READ|PROT_WRITE would
>> have changed anything here, but on *some* OSes it does, however it is
>> not portable.  At least OpenBSD doesn't behave like what you describe.
>
> Are you sure?  Can you point to concrete evidence?

Erm, I think there was a problem with my tests.

data:
- system has 8GB of ram
- no swap
- "data" rlimit set to 32GB, the per-process maximum supported on
  OpenBSD/amd64

with Daniel's test program asking for 20GB:
- mmap(PROT_NONE) _succeeds_
- mprotect(PROT_READ|PROT_WRITE) _succeeds_

An mmap call directly asking for 20GB with PROT_READ|PROT_WRITE also
succeeds.  The protection flags aren't checked to decide whether ENOMEM
should be returned, and the process has no easy way to tell whether the
requested amount of memory is actually usable (-> SIGBUS if the system
can't map enough pages).

The reason why my test initially failed is that I assumed that ulimit -d
was 4GB on this box, not 1.5GB (default for OpenBSD/amd64).  Not
double-checking this was sloppy, my sincere apologies to Daniel and the
other readers.

> Not that's it's important (using a hard-coded number like 2GB wouldn't
> work, so we'd more likely use something like w32heap.c's "pre-allocate
> double the size", which doesn't suffer from that problem anyway and
> still guarantees efficient behavior when growing a buffer progressively
> from 1B to 100GB).

Ack.  Note that the test above was using the maximum value for
ulimit -d; for the record, a single allocation of 2GB would be rejected
by default on all of our supported platforms.

-- 
jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF  DDCC 0DFA 74AE 1524 E7EE

Re: When should ralloc.c be used?

[Daniel Colascione]

On 10/28/2016 04:40 AM, Jérémie Courrèges-Anglas wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> Cc: monnier@iro.umontreal.ca, eggert@cs.ucla.edu, emacs-devel@gnu.org
>>> From: Daniel Colascione <dancol@dancol.org>
>>> Date: Fri, 28 Oct 2016 01:11:08 -0700
>>>
>>> Say I mmap (anonymously, for simplicity) a page PROT_NONE. After the
>>> initial mapping, that address space is unavailable for other uses. But
>>> because the page protections are PROT_NONE, my program has no legal
>>> right to access that page, so the OS doesn't have to guarantee that it
>                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>> can find a physical page to back that page I've mmaped. In this state,
>    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> This is what I think is a problem in your reasoning.  "Doesn't have to
> guarantee" doesn't mean that the kernel *should not* actually check the
> available memory and resource limits.
>

IMHO, an OS that rejects big PROT_NONE mappings merely because it might 
not be able to change them to PROT_READ|PROT_WRITE later is broken. The 
non-overcommit Linux behavior (which is identical to Windows behavior) 
is the _right _thing_ _to_ _do_. The OS is letting the process manage 
its address space and assuming that the programmer knows what he wanted 
to do.

>>> the memory is reserved.
>>>
>>> The 20GB PROT_NONE address space reservation itself requires very little
>>> memory. It's just a note in the kernel's VM interval tree that says "the
>>> addresses in range [0x20000, 0x500020000) are reserved". Virtual memory is
>>>
>>> Now imagine I change the protections to PROT_READ|PROT_WRITE --- once
>>> the PROT_READ|PROT_WRITE mprotect succeeds, my program has every right
>>> to access that page; under a strict accounting scheme (that is, without
>>> overcommit), the OS has to guarantee that it'll be able to go find a
>>> physical page to back that virtual page. In this state, the memory is
>>> committed -- the kernel has committed to finding backing storage for
>>> that page at some point when the current process tries to access it.
>>
>> I'm with you up to here.  My question is whether PROT_READ|PROT_WRITE
>> call could fail after PROT_NONE succeeded.  You seem to say it could;
>> I thought it couldn't.
>
> I wouldn't have thought that PROT_NONE vs PROT_READ|PROT_WRITE would
> have changed anything here, but on *some* OSes it does, however it is
> not portable.  At least OpenBSD doesn't behave like what you describe.

How does it behave?

> IMHO people who rely on this kind of reservations rely on
> implementation-defined behavior.

OpenBSD is a Coelacanth. It's a relic. It doesn't even a unified buffe 
cache.

> Also, sanity wise, I'd prefer having mmap(2) fail right away rather than
> having mprotect(2) fail, much later.

Then ask for PROT_READ|PROT_WRITE access right away. Ask for commit, not 
just address space.

> *If* mprotect(2) actually fails ;
> of course, you don't want to play russian roulette with your OS's
> flavor of the OOM-killer either.

That's why overcommit is an abomination.

Re: When should ralloc.c be used?

[Stefan Monnier]

>> > Using mmap has disadvantages: when you need to enlarge buffer text,
>> > and that fails (because there are no more free pages/addresses after
>> > the already allocated region), we need to copy buffer text to the new
>> > allocation.
>> All allocators suffer from this problem.  I haven't seen any evidence
>> that the mmap-based allocation code is significantly more prone to it.
> I have seen that.

Could you give some details (mostly about the scale of the problem)?

> The native glibc malloc, the on GNU/Linux systems were using until we
> got screwed by the recent glibc, didn't have this problem, while
> mmap-based allocator did.  Don't ask me how glibc does it, I don't
> know; but the fact is there.

It likely mmaps a bit more than requested, like you do in w32heap.c.

>> Another advantage of using mmap is that it can return the memory to the
>> OS once you kill your large buffer, whereas with gmalloc+ralloc this
>> basically never happens, AFAIK.
> Not entirely true: ralloc calls the system sbrk with a negative
> argument when it feels like it.

That's why I said "basically".  Yes, in theory it can sometimes
return memory.  In practice, this is rare.  In contrast, with mmap,
returning memory to the OS is the rule rather than the exception.


        Stefan

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: eggert@cs.ucla.edu, emacs-devel@gnu.org
> Date: Mon, 24 Oct 2016 14:45:59 -0400
> 
> >> > Using mmap has disadvantages: when you need to enlarge buffer text,
> >> > and that fails (because there are no more free pages/addresses after
> >> > the already allocated region), we need to copy buffer text to the new
> >> > allocation.
> >> All allocators suffer from this problem.  I haven't seen any evidence
> >> that the mmap-based allocation code is significantly more prone to it.
> > I have seen that.
> 
> Could you give some details (mostly about the scale of the problem)?

Visiting a large compressed file (e.g., an Emacs release tarball
compressed with gzip) takes with mmap several times as long as in a
build without mmap.

> > Not entirely true: ralloc calls the system sbrk with a negative
> > argument when it feels like it.
> 
> That's why I said "basically".  Yes, in theory it can sometimes
> return memory.  In practice, this is rare.  In contrast, with mmap,
> returning memory to the OS is the rule rather than the exception.

How so?  Releasing memory in both cases requires basically the same
situation: a large enough block of contiguous memory not in use.  It
seems ralloc is actually at an advantage, because relocating blocks
helps collect together a larger free block.

Re: When should ralloc.c be used?

[Stefan Monnier]

>> That's why I said "basically".  Yes, in theory it can sometimes
>> return memory.  In practice, this is rare.  In contrast, with mmap,
>> returning memory to the OS is the rule rather than the exception.
> How so?  Releasing memory in both cases requires basically the same
> situation: a large enough block of contiguous memory not in use.

IIUC releasing memory with sbrk can only be done if that memory is at
the end of the heap.

> It seems ralloc is actually at an advantage, because relocating blocks
> helps collect together a larger free block.

mmap can always free what it has allocated before, without any need to
relocate anything.


        Stefan

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 25 Oct 2016 10:12:23 -0400
> 
> >> That's why I said "basically".  Yes, in theory it can sometimes
> >> return memory.  In practice, this is rare.  In contrast, with mmap,
> >> returning memory to the OS is the rule rather than the exception.
> > How so?  Releasing memory in both cases requires basically the same
> > situation: a large enough block of contiguous memory not in use.
> 
> IIUC releasing memory with sbrk can only be done if that memory is at
> the end of the heap.

Since ralloc.c relocates blocks, it can make this happen more easily.

> > It seems ralloc is actually at an advantage, because relocating blocks
> > helps collect together a larger free block.
> 
> mmap can always free what it has allocated before, without any need to
> relocate anything.

It makes no sense to release random pages here and there, all you get
is fragmentation at address space level.

Re: When should ralloc.c be used?

[Stefan Monnier]

>> IIUC releasing memory with sbrk can only be done if that memory is at
>> the end of the heap.
> Since ralloc.c relocates blocks, it can make this happen more easily.

But the sbrk area is shared with gmalloc, whose data is not relocatable,
so as soon as gmalloc calls sbrk, the space previously allocated by
ralloc can't be returned any more.

>> > It seems ralloc is actually at an advantage, because relocating blocks
>> > helps collect together a larger free block.
>> mmap can always free what it has allocated before, without any need to
>> relocate anything.
> It makes no sense to release random pages here and there, all you get
> is fragmentation at address space level.

Yes, but address space is much more plentiful.

Note that glibc uses exactly this approach and it works very well for us.


        Stefan

Re: When should ralloc.c be used?

[Ken Raeburn]

On Oct 24, 2016, at 09:05, Eli Zaretskii <eliz@gnu.org> wrote:

> Using mmap has disadvantages: when you need to enlarge buffer text,
> and that fails (because there are no more free pages/addresses after
> the already allocated region), we need to copy buffer text to the new
> allocation.  This happens quite a lot when we visit a compressed
> buffer.  (The MS-Windows emulation of mmap in w32heap.c reserves twice
> the number of pages as originally requested, for that very reason.)

In the general case, yes.  But modern Linux kernels have an “mremap” system call which can “move” a range of pages to a portion of the address space that can accommodate a larger size, by tweaking page tables rather than copying all the bits around.  I’m pretty sure modern glibc realloc uses it.  I had a project a while back where code ported to Solaris ran far slower than the GNU/Linux version because lots of realloc calls were done on a large array; Solaris copied, GNU/Linux remapped.

  void *mremap(void *old_address, size_t old_size,
               size_t new_size, int flags, ... /* void *new_address */);

Of course you can’t shift bytes within a page this way, or add new space anywhere but after the last page of the old region.  (No hint in the man page whether you can use an explicit new address range overlapping the old range to shift a chunk of memory a la memmove, or if the results would be undefined a la memcpy.)

I don’t know if any other systems support it.  The performance savings for one of our favorite systems might be worth the special-casing.  Though, if glibc realloc does the right thing, maybe using malloc/realloc for buffer storage would suffice.

Ken

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Mon, 24 Oct 2016 23:12:40 -0400
> 
> > Using mmap has disadvantages: when you need to enlarge buffer text,
> > and that fails (because there are no more free pages/addresses after
> > the already allocated region), we need to copy buffer text to the new
> > allocation.  This happens quite a lot when we visit a compressed
> > buffer.  (The MS-Windows emulation of mmap in w32heap.c reserves twice
> > the number of pages as originally requested, for that very reason.)
> 
> In the general case, yes.  But modern Linux kernels have an “mremap” system 
> call which can “move” a range of pages to a portion of the address space that 
> can accommodate a larger size, by tweaking page tables rather than copying all 
> the bits around.  I’m pretty sure modern glibc realloc uses it.

AFAIU, this feature will only help us if someone adds code to use it
in buffer.c:mmap_enlarge.  Or are you saying that the OS will call
mremap for us automatically when mmap_enlarge attempts to map
additional pages at the end of an mmaped region?

> I don’t know if any other systems support it.  The performance savings for one 
> of our favorite systems might be worth the special-casing.  Though, if glibc 
> realloc does the right thing, maybe using malloc/realloc for buffer storage 
> would suffice.

If the Linux kernel is the only system that allows implementation of
mremap, then it doesn't really help in the long run, because on master
we don't need mmap at all for GNU/Linux systems.

Re: When should ralloc.c be used?

[Ken Raeburn]

> On Oct 25, 2016, at 12:06, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Ken Raeburn <raeburn@raeburn.org>
>> Date: Mon, 24 Oct 2016 23:12:40 -0400
>> 
>>> Using mmap has disadvantages: when you need to enlarge buffer text,
>>> and that fails (because there are no more free pages/addresses after
>>> the already allocated region), we need to copy buffer text to the new
>>> allocation.  This happens quite a lot when we visit a compressed
>>> buffer.  (The MS-Windows emulation of mmap in w32heap.c reserves twice
>>> the number of pages as originally requested, for that very reason.)
>> 
>> In the general case, yes.  But modern Linux kernels have an “mremap” system 
>> call which can “move” a range of pages to a portion of the address space that 
>> can accommodate a larger size, by tweaking page tables rather than copying all 
>> the bits around.  I’m pretty sure modern glibc realloc uses it.
> 
> AFAIU, this feature will only help us if someone adds code to use it
> in buffer.c:mmap_enlarge.  Or are you saying that the OS will call
> mremap for us automatically when mmap_enlarge attempts to map
> additional pages at the end of an mmaped region?

It could be done explicitly, but my experience was that malloc/realloc would just do it for us; we’d just have to use malloc/realloc instead of explicitly calling mmap.  I just took a quick look at the glibc sources (2.19, as patched and packaged by Debian), and it looks like the use of mmap kicks in by default for 128kB or larger allocations, though the threshold can be changed at run time.

> If the Linux kernel is the only system that allows implementation of
> mremap, then it doesn't really help in the long run, because on master
> we don't need mmap at all for GNU/Linux systems.

A man page browser at freebsd.org for several platforms seems to indicate that NetBSD has picked it up, but neither FreeBSD nor OpenBSD.  I don’t know if NetBSD’s realloc will use it, but it’s certainly simpler if we just ignore mremap for explicit use, and just bear in mind that realloc may not always have to pay the expected copying penalty on all systems….

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Ken Raeburn <raeburn@raeburn.org>
> Date: Wed, 26 Oct 2016 00:36:42 -0400
> Cc: emacs-devel@gnu.org
> 
> >>> Using mmap has disadvantages: when you need to enlarge buffer text,
> >>> and that fails (because there are no more free pages/addresses after
> >>> the already allocated region), we need to copy buffer text to the new
> >>> allocation.  This happens quite a lot when we visit a compressed
> >>> buffer.  (The MS-Windows emulation of mmap in w32heap.c reserves twice
> >>> the number of pages as originally requested, for that very reason.)
> >> 
> >> In the general case, yes.  But modern Linux kernels have an “mremap” system 
> >> call which can “move” a range of pages to a portion of the address space 
> >> that 
> >> can accommodate a larger size, by tweaking page tables rather than copying 
> >> all 
> >> the bits around.  I’m pretty sure modern glibc realloc uses it.
> > 
> > AFAIU, this feature will only help us if someone adds code to use it
> > in buffer.c:mmap_enlarge.  Or are you saying that the OS will call
> > mremap for us automatically when mmap_enlarge attempts to map
> > additional pages at the end of an mmaped region?
> 
> It could be done explicitly, but my experience was that malloc/realloc would 
> just do it for us; we’d just have to use malloc/realloc instead of explicitly 
> calling mmap.

I think we've lost context of the discussion.  Please see above: this
is about the disadvantages of using mmap directly, i.e. for those
cases where the native malloc or gmalloc suffer from memory
fragmentation, and we decide to use mmap in buffer.c to countermand
that.

I've pointed out the disadvantages of using mmap directly, and you
mentioned the mremap syscall as the counter-argument.  If you thought
I was talking about problems mmap could cause to the malloc
implementation, then that's a misunderstanding: I was explicitly
talking about using mmap directly for allocating buffer text.  My
point was that we should only use mmap if necessary, as it comes for a
price.

> A man page browser at freebsd.org for several platforms seems to indicate that 
> NetBSD has picked it up, but neither FreeBSD nor OpenBSD.  I don’t know if 
> NetBSD’s realloc will use it, but it’s certainly simpler if we just ignore 
> mremap for explicit use, and just bear in mind that realloc may not always have 
> to pay the expected copying penalty on all systems….

Once again, this is about the cases where using malloc for buffer text
gives unsatisfactory results, and mmap is being considered as a
remedy.

Re: When should ralloc.c be used?

[Ken Raeburn]

> I think we've lost context of the discussion.  Please see above: this
> is about the disadvantages of using mmap directly, i.e. for those
> cases where the native malloc or gmalloc suffer from memory
> fragmentation, and we decide to use mmap in buffer.c to countermand
> that.

Yes, sorry, I got a bit off track…

Ken

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 23 Oct 2016 16:44:10 -0400
> 
> >> I don't think it matters very much since we use mmap for the buffers,
> > No, we don't, not on GNU/Linux anyway.
> 
> AFAIK the decision not to use mmap was due to the fact that glibc's
> malloc itself uses mmap.  But if we don't use glibc's malloc, then why
> wouldn't we decide to use mmap ourselves for the buffers?

I already asked that:

  http://lists.gnu.org/archive/html/emacs-devel/2016-10/msg00678.html

The only answer was disappointing:

  I don't know, and would rather not spend time investigating.

It looks like either people don't realize what a land mine we just
stepped on, or they simply don't care enough.

Does it make sense to anyone to release Emacs 25.2 that doesn't work
reliably on recent GNU/Linux systems?  Because that's what is going to
happen if we don't invest all the resources we have into solving this.

Re: When should ralloc.c be used?

[Stefan Monnier]

> It looks like either people don't realize what a land mine we just
> stepped on, or they simply don't care enough.
[...]
> Does it make sense to anyone to release Emacs 25.2 that doesn't work
> reliably on recent GNU/Linux systems?  Because that's what is going to
> happen if we don't invest all the resources we have into solving this.

I must be missing something: you seem to know about past severe problems
we've had because of fragmentation, whereas I can't remember any
such occurrence.


        Stefan

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Mon, 24 Oct 2016 08:45:04 -0400
> 
> > It looks like either people don't realize what a land mine we just
> > stepped on, or they simply don't care enough.
> [...]
> > Does it make sense to anyone to release Emacs 25.2 that doesn't work
> > reliably on recent GNU/Linux systems?  Because that's what is going to
> > happen if we don't invest all the resources we have into solving this.
> 
> I must be missing something: you seem to know about past severe problems
> we've had because of fragmentation, whereas I can't remember any
> such occurrence.

I don't understand how you get to talking about fragmentation.  I
never mentioned anything like that.  The problems I was talking about
are all related to using ralloc.c on GNU/Linux systems with a recent
enough glibc.

Re: When should ralloc.c be used?

[Stefan Monnier]

> I don't understand how you get to talking about fragmentation.  I
> never mentioned anything like that.  The problems I was talking about
> are all related to using ralloc.c on GNU/Linux systems with a recent
> enough glibc.

I misunderstood, then.  I fully agree that ralloc.c is a landmine, if
that's what you meant.  That's why I think we should get rid of it.

And if we really want to keep it, we should prefer mmap over ralloc
(i.e. we should only consider ralloc in those cases where the mmap
alternative is unavailable (not sure if there are still systems where
this is the case, the DOS port maybe?)).

AFAIK, gmalloc+mmap-ralloc is a perfectly acceptable solution for
Emacs-25.2 with the new glibc, with no known problem.


        Stefan

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Mon, 24 Oct 2016 10:42:02 -0400
> 
> AFAIK, gmalloc+mmap-ralloc is a perfectly acceptable solution for
> Emacs-25.2 with the new glibc, with no known problem.

So you consider this preferable to the 2 alternatives I mentioned in

  http://lists.gnu.org/archive/html/emacs-devel/2016-10/msg00740.html

?

They both avoid using mmap, since I won't want to re-introduce its
disadvantages to GNU/Linux systems.

Re: When should ralloc.c be used?

[Stefan Monnier]

>> AFAIK, gmalloc+mmap-ralloc is a perfectly acceptable solution for
>> Emacs-25.2 with the new glibc, with no known problem.
> So you consider this preferable to the 2 alternatives I mentioned in
>   http://lists.gnu.org/archive/html/emacs-devel/2016-10/msg00740.html
> ?

Not sure.  AFAIU, gmalloc-mmap-ralloc suffers from fragmentation, which
was the reason why ralloc was written in the first place, so I would
tend to shy away from it, but I have not personally seen those problems,
so I don't have a strong opinion on this.

As for using HYBRID_MALLOC, that would be a better solution I think, but
I haven't looked at the corresponding patch, so I don't know how safe it is.


        Stefan

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 24 Oct 2016 10:42:02 -0400
> Cc: emacs-devel@gnu.org
> 
> And if we really want to keep it, we should prefer mmap over ralloc
> (i.e. we should only consider ralloc in those cases where the mmap
> alternative is unavailable (not sure if there are still systems where
> this is the case, the DOS port maybe?)).

The DOS port has code to work with its system malloc.  That code was
tested at the time, so this port shouldn't be an obstacle on the way
of getting rid of ralloc.c.

> AFAIK, gmalloc+mmap-ralloc is a perfectly acceptable solution for
> Emacs-25.2 with the new glibc, with no known problem.

I think gmalloc without mmap might be better.  See the alternatives I
mentioned elsewhere in this thread.

Re: When should ralloc.c be used?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I must be missing something: you seem to know about past severe problems
  > we've had because of fragmentation, whereas I can't remember any
  > such occurrence.

Fragmentation caused problems bad enough to motivate me to write
ralloc.  But I don't know about the present situation.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I don't like it either, but would rather work on redoing the build process so 
  > that we can use the native malloc on all hosts.

That may not be desirable, though.  We started using GNU malloc
because it gave much better performance than some native mallocs.
Whether that is true today, I have no idea; I am only saying
that it is an issue to consider.

Stefan said:

  > But that doesn't explain why we'd need to use ralloc in the mean time.

Why would we not want to use ralloc?  It made a big improvement for
memory management when I wrote it.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Paul Eggert]

Richard Stallman wrote:
>   > I don't like it either, but would rather work on redoing the build process so
>   > that we can use the native malloc on all hosts.
>
> That may not be desirable, though.  We started using GNU malloc
> because it gave much better performance

We could continue to do that, on the set of platforms where our copy of GNU 
malloc works significantly better than native malloc. My impression, though, is 
that this set of platforms is gradually shrinking due to improvements in memory 
allocators. See, e.g.:

Berger ED, Zorn BG, McKinley KS. Reconsidering custom memory allocation. 
OOPSLA'02. http://dx.doi.org/10.1145/2502508.2502522
https://people.cs.umass.edu/~emery/pubs/berger-oopsla2002.pdf

Re: When should ralloc.c be used? (WAS: bug#24358)

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: eliz@gnu.org, emacs-devel@gnu.org, npostavs@users.sourceforge.net
> Date: Sun, 23 Oct 2016 20:21:33 -0400
> 
>   > I don't like it either, but would rather work on redoing the build process so 
>   > that we can use the native malloc on all hosts.
> 
> That may not be desirable, though.  We started using GNU malloc
> because it gave much better performance than some native mallocs.
> Whether that is true today, I have no idea; I am only saying
> that it is an issue to consider.

I think native malloc on GNU/Linux is much better these days; we were
using it all the recent years, until glibc developers removed the
hooks we needed for unexec support (which is why those GNU/Linux
systems where this change is already installed switched to gmalloc and
ralloc instead).

Emacs 25.1 switched to native malloc on MS-Windows as well, and I see
no problems with memory management due to that, perhaps even a small
improvement.

>   > But that doesn't explain why we'd need to use ralloc in the mean time.
> 
> Why would we not want to use ralloc?

It imposes hard-to-fulfill requirements on functions that get C
pointers to buffer text or to Lisp string data: those functions must
never call malloc, directly or indirectly.

This requirement was well known to the few Emacs developers in the
distant past, when all the platforms used ralloc.  But since the
modern platforms gradually migrated away from ralloc, this is almost
unknown to most current developers, and code crept in that violates
this requirement.  Fixing all that code is hard, because most of it is
not easily found; it manifests itself in corruption of buffer text,
random segfaults and aborts during GC, which happen long time after
the offending code did its job.

> It made a big improvement for memory management when I wrote it.

It is no longer a big improvement, as modern platforms manage memory
much better in their native malloc implementations.  So ralloc is
nowadays a significant disadvantage almost without advantages.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I think native malloc on GNU/Linux is much better these days; we were
  > using it all the recent years, until glibc developers removed the
  > hooks we needed for unexec support (which is why those GNU/Linux
  > systems where this change is already installed switched to gmalloc and
  > ralloc instead).

Should we talk with them about putting in those hooks or other
suitable hooks?  Then we could go back to the libc malloc.

  > It imposes hard-to-fulfill requirements on functions that get C
  > pointers to buffer text or to Lisp string data: those functions must
  > never call malloc, directly or indirectly.

I think the way to fix those is by systematically looking at the
source for them, rather than by debugging.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: eggert@cs.ucla.edu, emacs-devel@gnu.org,
> 	npostavs@users.sourceforge.net
> Date: Mon, 24 Oct 2016 12:55:36 -0400
> 
>   > I think native malloc on GNU/Linux is much better these days; we were
>   > using it all the recent years, until glibc developers removed the
>   > hooks we needed for unexec support (which is why those GNU/Linux
>   > systems where this change is already installed switched to gmalloc and
>   > ralloc instead).
> 
> Should we talk with them about putting in those hooks or other
> suitable hooks?  Then we could go back to the libc malloc.

I think we tried, and more or less failed.  (That was in the context
of unexec, but the arguments are more or less similar.)

>   > It imposes hard-to-fulfill requirements on functions that get C
>   > pointers to buffer text or to Lisp string data: those functions must
>   > never call malloc, directly or indirectly.
> 
> I think the way to fix those is by systematically looking at the
> source for them, rather than by debugging.

Yes, but finding out whether this is so is not easy, because the
malloc call is sometimes buried very deep.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Should we talk with them about putting in those hooks or other
  > > suitable hooks?  Then we could go back to the libc malloc.

  > I think we tried, and more or less failed.  (That was in the context
  > of unexec, but the arguments are more or less similar.)

How did it fail?  Did they give it a strong try?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Paul Eggert]

Richard Stallman wrote:
>   > > Should we talk with them about putting in those hooks or other
>   > > suitable hooks?  Then we could go back to the libc malloc.
>
>   > I think we tried, and more or less failed.  (That was in the context
>   > of unexec, but the arguments are more or less similar.)
>
> How did it fail?  Did they give it a strong try?

It was more the other way around. People working on the glibc memory allocator 
convinced me that the malloc hooks were a significant impediment to performance 
improvements within glibc, and that Emacs unexec didn't really need those hooks 
any more. Emacs was the only major user of that part of the old glibc API.

For those interested in GNU malloc performance improvements, a talk related to 
the current effort is scheduled a week from Thursday in Santa Fe. Please see:

O'Donell C. linux and glibc: The 4.5TiB malloc API trace. LPC 2016. 
https://linuxplumbersconf.org/2016/ocw/proposals/3921

Re: When should ralloc.c be used? (WAS: bug#24358)

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: eggert@cs.ucla.edu, emacs-devel@gnu.org,
> 	npostavs@users.sourceforge.net
> Date: Mon, 24 Oct 2016 22:35:53 -0400
> 
>   > > Should we talk with them about putting in those hooks or other
>   > > suitable hooks?  Then we could go back to the libc malloc.
> 
>   > I think we tried, and more or less failed.  (That was in the context
>   > of unexec, but the arguments are more or less similar.)
> 
> How did it fail?

My take is that the glibc developers don't really want to hear about
keeping those hooks.

> Did they give it a strong try?

I don't know what that means in practice.  What would make the try
"strong"?

You can see the discussion starting here:

  http://lists.gnu.org/archive/html/emacs-devel/2016-01/msg00956.html

You took some part in the discussion, at least its public part (I
understand there was also an off-list part).  I think once you said
here:

  http://lists.gnu.org/archive/html/emacs-devel/2016-01/msg01633.html

that you favored replacing unexec by a more portable scheme, there was
no longer any reasons to make our argument stronger.

Since then Paul implemented a workaround on the master branch, which
uses gmalloc during dumping, and switches to the native malloc in the
dumped executable.

At the time, we didn't realize, I think, that removing the glibc hooks
will cause GNU/Linux systems to start using ralloc.c, which is the
trigger for the present discussion.  The discovery of this issue means
that the hope expressed in the Jan 2016 discussions that Emacs
versions before 25 will continue to be usable on GNU/Linux systems
with a newer glibc -- that hope was too optimistic.  Based on what
we've learned the hard way during the last couple of weeks, I'd say
that all the Emacs versions before 25.2 (including 25.1) will be
unstable on such GNU systems to the degree of making them almost
unusable.  E.g., one bug report related to this claims crashes inside
GC once every 10-15 minutes, something that is IMO unbearably
frequent, especially given that segfaults during GC almost always
cause loss of work (because auto-saving almost always fails).

Re: When should ralloc.c be used? (WAS: bug#24358)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   I think once you said
  > here:

  >   http://lists.gnu.org/archive/html/emacs-devel/2016-01/msg01633.html

  > that you favored replacing unexec by a more portable scheme, there was
  > no longer any reasons to make our argument stronger.

In general, I'm in favor of a more portable method.  But we don't have
one now.  Is it feasible to do?  Is anyone working on one?  

If not, then I hope we can design, with the Glibc developers, a
different set of hooks to allow us to make unexec work.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Paul Eggert]

Richard Stallman wrote:
> In general, I'm in favor of a more portable method.  But we don't have
> one now.  Is it feasible to do?  Is anyone working on one?

Yes, it's feasible. It is on my list of things of do. Admittedly I'm stretched 
thin, and the approach I prefer (generating and then compiling C code) is not 
everybody's favorite.

> If not, then I hope we can design, with the Glibc developers, a
> different set of hooks to allow us to make unexec work.

That would be more work than fixing Emacs, I expect. Plus, malloc hooks are not 
the only reason  unexec is dicey.

> We should withdraw 25.1, I think.

I don't think that will help. Similar problems likely affect 24.5 and earlier 
versions, if they are built against bleeding-edge glibc and are configured in 
the default way.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Eli Zaretskii]

> Cc: npostavs@users.sourceforge.net, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 25 Oct 2016 22:08:21 -0700
> 
> > We should withdraw 25.1, I think.
> 
> I don't think that will help. Similar problems likely affect 24.5 and earlier 
> versions, if they are built against bleeding-edge glibc and are configured in 
> the default way.

Indeed, I agree.  People who first bump into this with Emacs 25.1 will
have 25.2 soon enough (I hope).  By contrast, those who will try to
build Emacs 24.x on the newer GNU/Linux systems will be unable to
resolve the instability problems exposed by using ralloc.c, except by
back-porting patches we have just committed to the Emacs repository,
which is not easy.

Not sure what to do with the old versions, or whether anything can be
done.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Noam Postavsky]

On Wed, Oct 26, 2016 at 7:46 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Cc: npostavs@users.sourceforge.net, emacs-devel@gnu.org
>> From: Paul Eggert <eggert@cs.ucla.edu>
>> Date: Tue, 25 Oct 2016 22:08:21 -0700
>>
>> > We should withdraw 25.1, I think.
>>
>> I don't think that will help. Similar problems likely affect 24.5 and earlier
>> versions, if they are built against bleeding-edge glibc and are configured in
>> the default way.
>
> Indeed, I agree.  People who first bump into this with Emacs 25.1 will
> have 25.2 soon enough (I hope).  By contrast, those who will try to
> build Emacs 24.x on the newer GNU/Linux systems will be unable to
> resolve the instability problems exposed by using ralloc.c, except by
> back-porting patches we have just committed to the Emacs repository,
> which is not easy.

Wouldn't configuring with REL_ALLOC=no work?

Re: When should ralloc.c be used? (WAS: bug#24358)

[Eli Zaretskii]

> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Date: Wed, 26 Oct 2016 09:10:35 -0400
> Cc: Paul Eggert <eggert@cs.ucla.edu>, rms@gnu.org, Emacs developers <emacs-devel@gnu.org>
> 
> Wouldn't configuring with REL_ALLOC=no work?

It could, yes.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Yes, it's feasible. It is on my list of things of do. Admittedly I'm stretched 
  > thin, and the approach I prefer (generating and then compiling C code) is not 
  > everybody's favorite.

Could you explain that more?

Does anyone want to implement another approach?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Paul Eggert]

On 10/26/2016 06:23 PM, Richard Stallman wrote:
> Could you explain that more?

The main idea is to save the current Emacs state as C source code, then 
compile the (large and boring) .c file and relink Emacs with the 
resulting .o file instead of a dummy .o file that it would start off 
with. Most of this new .o file would be data; perhaps some would be code 
that would initialize the data, though we'd want to minimize this.

> Does anyone want to implement another approach?

Eli has mentioned a simpler approach, where we build an .elc file when 
saving Emacs state and load the .elc file during normal startup. The 
main worry about this approach is performance.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Perry E. Metzger]

On Wed, 26 Oct 2016 18:36:02 -0700 Paul Eggert <eggert@cs.ucla.edu>
wrote:
> On 10/26/2016 06:23 PM, Richard Stallman wrote:
> > Could you explain that more?  
> 
> The main idea is to save the current Emacs state as C source code,
> then compile the (large and boring) .c file and relink Emacs with
> the resulting .o file instead of a dummy .o file that it would
> start off with. Most of this new .o file would be data; perhaps
> some would be code that would initialize the data, though we'd want
> to minimize this.

Could the new dynamic loading feature be used here so it wouldn't be
necessary to re-link emacs?

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: When should ralloc.c be used? (WAS: bug#24358)

[Paul Eggert]

On 10/27/2016 06:35 AM, Perry E. Metzger wrote:
> Could the new dynamic loading feature be used here so it wouldn't be necessary to re-link emacs?

It might be doable, though I expect it'd be more work. Dynamic loading 
purposely isolates modules from Emacs internals, and most likely we'd 
need several bridges over that moat.

On 10/27/2016 06:44 AM, Fabrice Popineau wrote:

> I find it disturbing that a C compiler will be needed to redump Emacs.
It's a tradeoff, yes. My impression is that the rare user who redumps 
Emacs typically has a C compiler installed or can easily install one, so 
it shouldn't be much to ask.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Perry E. Metzger]

On Thu, 27 Oct 2016 07:51:37 -0700 Paul Eggert <eggert@cs.ucla.edu>
wrote:
> On 10/27/2016 06:44 AM, Fabrice Popineau wrote:
> 
> > I find it disturbing that a C compiler will be needed to redump
> > Emacs.  
> It's a tradeoff, yes. My impression is that the rare user who
> redumps Emacs typically has a C compiler installed or can easily
> install one, so it shouldn't be much to ask.

Agreed. On free operating systems it's easy (that's the whole point of
freedom!), and even on non-free operating systems free compilers are
available, so it isn't a big deal for most users I think. (On macOS
the official XCode compiler is also available gratis.)

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: When should ralloc.c be used? (WAS: bug#24358)

[Eli Zaretskii]

> Date: Thu, 27 Oct 2016 11:05:03 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: rms@gnu.org, eliz@gnu.org, emacs-devel@gnu.org,
>  npostavs@users.sourceforge.net, Fabrice Popineau
>  <fabrice.popineau@gmail.com>
> 
> > It's a tradeoff, yes. My impression is that the rare user who
> > redumps Emacs typically has a C compiler installed or can easily
> > install one, so it shouldn't be much to ask.
> 
> Agreed. On free operating systems it's easy (that's the whole point of
> freedom!), and even on non-free operating systems free compilers are
> available, so it isn't a big deal for most users I think. (On macOS
> the official XCode compiler is also available gratis.)

I can assure you that installing a fully functioning environment for
compiling programs is not a trivial task on MS-Windows.  It isn't
enough to have just a compiler: you need Binutils, support libraries
and header files, and a well-configured MSYS installation to be able
to run Emacs build script and Makefiles.

Also, please don't forget that some people run Emacs on machines where
they are not system administrators and are not allowed to install
arbitrary packages.

Not everyone is in the same position as you and Paul (or myself).

Re: When should ralloc.c be used? (WAS: bug#24358)

[Perry E. Metzger]

On Thu, 27 Oct 2016 21:13:05 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Thu, 27 Oct 2016 11:05:03 -0400
> > From: "Perry E. Metzger" <perry@piermont.com>
> > Cc: rms@gnu.org, eliz@gnu.org, emacs-devel@gnu.org,
> >  npostavs@users.sourceforge.net, Fabrice Popineau
> >  <fabrice.popineau@gmail.com>
> >   
> > > It's a tradeoff, yes. My impression is that the rare user who
> > > redumps Emacs typically has a C compiler installed or can easily
> > > install one, so it shouldn't be much to ask.  
> > 
> > Agreed. On free operating systems it's easy (that's the whole
> > point of freedom!), and even on non-free operating systems free
> > compilers are available, so it isn't a big deal for most users I
> > think. (On macOS the official XCode compiler is also available
> > gratis.)  
> 
> I can assure you that installing a fully functioning environment for
> compiling programs is not a trivial task on MS-Windows.  It isn't
> enough to have just a compiler: you need Binutils, support libraries
> and header files, and a well-configured MSYS installation to be able
> to run Emacs build script and Makefiles.
> 
> Also, please don't forget that some people run Emacs on machines
> where they are not system administrators and are not allowed to
> install arbitrary packages.
> 
> Not everyone is in the same position as you and Paul (or myself).
> 

Sure, but most people never, ever undump an Emacs either unless
they're building from scratch or doing Emacs dev work...

-- 
Perry E. Metzger		perry@piermont.com

Re: When should ralloc.c be used? (WAS: bug#24358)

[Daniel Colascione]

On 10/27/2016 02:03 PM, Perry E. Metzger wrote:
> On Thu, 27 Oct 2016 21:13:05 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
>>> Date: Thu, 27 Oct 2016 11:05:03 -0400
>>> From: "Perry E. Metzger" <perry@piermont.com>
>>> Cc: rms@gnu.org, eliz@gnu.org, emacs-devel@gnu.org,
>>>  npostavs@users.sourceforge.net, Fabrice Popineau
>>>  <fabrice.popineau@gmail.com>
>>>
>>>> It's a tradeoff, yes. My impression is that the rare user who
>>>> redumps Emacs typically has a C compiler installed or can easily
>>>> install one, so it shouldn't be much to ask.
>>>
>>> Agreed. On free operating systems it's easy (that's the whole
>>> point of freedom!), and even on non-free operating systems free
>>> compilers are available, so it isn't a big deal for most users I
>>> think. (On macOS the official XCode compiler is also available
>>> gratis.)
>>
>> I can assure you that installing a fully functioning environment for
>> compiling programs is not a trivial task on MS-Windows.  It isn't
>> enough to have just a compiler: you need Binutils, support libraries
>> and header files, and a well-configured MSYS installation to be able
>> to run Emacs build script and Makefiles.
>>
>> Also, please don't forget that some people run Emacs on machines
>> where they are not system administrators and are not allowed to
>> install arbitrary packages.
>>
>> Not everyone is in the same position as you and Paul (or myself).
>>
>
> Sure, but most people never, ever undump an Emacs either unless
> they're building from scratch or doing Emacs dev work...
>

That's because it doesn't really work. That's why I added code that 
explicitly stops repeated dumps. It doesn't mean I don't want to support 
user dumps.

It pains me to see people tolerate 30 second Emacs startup times. The 
daemon is a hack. I want Emacs normal mode of operation to be to start 
from *user* *specific* saved state --- that way, all Emacs instances can 
be as fast as emacs -Q.

If we need a compiler to make this happen, so be it. We'll just require 
libgcc, or hell, check it in to the repository, the way gcc checks in 
its dependencies.

An additional benefit of integrating with a compiler at runtime is the 
potential to JIT elisp code. Both LLVM andGCC these days have usable JIT 
interfaces. We could even serialize JIT traces in these user Emacs dumps.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Perry E. Metzger]

On Thu, 27 Oct 2016 14:07:46 -0700 Daniel Colascione
<dancol@dancol.org> wrote:
> If we need a compiler to make this happen, so be it. We'll just
> require libgcc, or hell, check it in to the repository, the way gcc
> checks in its dependencies.
> 
> An additional benefit of integrating with a compiler at runtime is
> the potential to JIT elisp code. Both LLVM and GCC these days have
> usable JIT interfaces. We could even serialize JIT traces in these
> user Emacs dumps.

Having a JIT for emacs bytecode (or some other IR) would be really
superb. I had no idea that GCC now had JIT support, but if it is as
easy to use as LLVM's, a prototype would not be a hard project. (I
presume RMS would insist on GCC as the basis.)

Of course, given that Emacs already byte compiles everything, maybe
going straight to machine code rather than the bytecode + JIT would
be good? Again, I don't know what GCC's infra is like, but if it is
as good as LLVM's that would be quite straightforward.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: When should ralloc.c be used?

[Daniel Colascione]

"Perry E. Metzger" <perry@piermont.com> writes:

> On Thu, 27 Oct 2016 14:07:46 -0700 Daniel Colascione
> <dancol@dancol.org> wrote:
>> If we need a compiler to make this happen, so be it. We'll just
>> require libgcc, or hell, check it in to the repository, the way gcc
>> checks in its dependencies.
>> 
>> An additional benefit of integrating with a compiler at runtime is
>> the potential to JIT elisp code. Both LLVM and GCC these days have
>> usable JIT interfaces. We could even serialize JIT traces in these
>> user Emacs dumps.
>
> Having a JIT for emacs bytecode (or some other IR) would be really
> superb. I had no idea that GCC now had JIT support, but if it is as
> easy to use as LLVM's, a prototype would not be a hard project. (I
> presume RMS would insist on GCC as the basis.)

GCC's interface isn't nearly as mature as LLVM's yet, but there's promise

https://gcc.gnu.org/wiki/JIT

> Of course, given that Emacs already byte compiles everything, maybe
> going straight to machine code rather than the bytecode + JIT would
> be good? Again, I don't know what GCC's infra is like, but if it is
> as good as LLVM's that would be quite straightforward.

AOT is all the rage right now (JEP 295), but I believe that tracing JITs
are ultimately the right choice for code density and installation
latency reasons. But this is one of those arguments that's never going
to be solved.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Eli Zaretskii]

> Cc: npostavs@users.sourceforge.net, eggert@cs.ucla.edu,
>  fabrice.popineau@gmail.com, rms@gnu.org, emacs-devel@gnu.org
> From: Daniel Colascione <dancol@dancol.org>
> Date: Thu, 27 Oct 2016 14:07:46 -0700
> 
> > Sure, but most people never, ever undump an Emacs either unless
> > they're building from scratch or doing Emacs dev work...
> >
> 
> That's because it doesn't really work. That's why I added code that 
> explicitly stops repeated dumps. It doesn't mean I don't want to support 
> user dumps.

Exactly.

> It pains me to see people tolerate 30 second Emacs startup times. The 
> daemon is a hack. I want Emacs normal mode of operation to be to start 
> from *user* *specific* saved state --- that way, all Emacs instances can 
> be as fast as emacs -Q.
> 
> If we need a compiler to make this happen, so be it.

But if there's a simpler method that doesn't get us enywhere near 30
sec, that should be "good enough".

> We'll just require libgcc, or hell, check it in to the repository,

You can't, not without having all the GCC sources.  But that's an
aside.

> An additional benefit of integrating with a compiler at runtime is the 
> potential to JIT elisp code. Both LLVM andGCC these days have usable JIT 
> interfaces. We could even serialize JIT traces in these user Emacs dumps.

This should be an opt-in feature, not a hard requirement, IMO.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Eli Zaretskii]

> Date: Thu, 27 Oct 2016 17:03:45 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: eggert@cs.ucla.edu, rms@gnu.org, emacs-devel@gnu.org,
>  npostavs@users.sourceforge.net, fabrice.popineau@gmail.com
> 
> > Not everyone is in the same position as you and Paul (or myself).
> > 
> 
> Sure, but most people never, ever undump an Emacs either unless
> they're building from scratch or doing Emacs dev work...

Oh, so now we are going to argue that a feature that can't be easily
had is not important?  Then I'll claim that the Emacs startup time is
not important, either, because "most people never, ever" start Emacs
except when their machine starts, and their Emacs session is
thereafter running for weeks and months without ever restarting.

Let's agree to respect other people's usage patterns and
circumstances, even if they are different from ours.  Emacs is great
because it allows so many different patterns, so preferring one of
them too much is something we should avoid.  If each one of us sees
only their personal needs as the most important ones, we will never be
able to agree on anything.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Fabrice Popineau]

[-- Attachment #1: Type: text/plain, Size: 603 bytes --]

2016-10-27 3:36 GMT+02:00 Paul Eggert <eggert@cs.ucla.edu>:

> On 10/26/2016 06:23 PM, Richard Stallman wrote:
>
>> Could you explain that more?
>>
>
> The main idea is to save the current Emacs state as C source code, then
> compile the (large and boring) .c file and relink Emacs with the resulting
> .o file instead of a dummy .o file that it would start off with. Most of
> this new .o file would be data; perhaps some would be code that would
> initialize the data, though we'd want to minimize this.
>
>
I find it disturbing that a C compiler will be needed to redump Emacs.
Am I alone ?

Fabrice

[-- Attachment #2: Type: text/html, Size: 1142 bytes --]

Re: When should ralloc.c be used? (WAS: bug#24358)

[Eli Zaretskii]

> From: Fabrice Popineau <fabrice.popineau@gmail.com>
> Date: Thu, 27 Oct 2016 15:44:27 +0200
> Cc: rms@gnu.org, Eli Zaretskii <eliz@gnu.org>, Emacs developers <emacs-devel@gnu.org>, 
> 	Noam Postavsky <npostavs@users.sourceforge.net>
> 
>  The main idea is to save the current Emacs state as C source code, then compile the (large and
>  boring) .c file and relink Emacs with the resulting .o file instead of a dummy .o file that it would start off
>  with. Most of this new .o file would be data; perhaps some would be code that would initialize the data,
>  though we'd want to minimize this.
> 
> I find it disturbing that a C compiler will be needed to redump Emacs.
> Am I alone ? 

No.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Eli has mentioned a simpler approach, where we build an .elc file when 
  > saving Emacs state and load the .elc file during normal startup. The 
  > main worry about this approach is performance.

There is no reason why we have to choose between C code and Lisp code.
It's worth developing a special-purpose format for this, if that would
be considerably faster.

However, reading a file that specifies construction of objects is
always going to be slower than copying a memory dump.  What part of the
time could we save with a different format?

Here's an idea.  We separate (1) creating objects from (2) storing them.
We define several operations.  We record all objects created
by sequence number.

We have these ways of creating an object.  That object is given
the next consecutive sequence number.

* intern (specify symbol name)
* variable value (specify variable name)
* string (specify contents)
* integer (specify value)
* cons (specify two sequence numbers)
* list (specify N sequence numbers)
* array (specify N sequence numbers)
* expression (specify the expression textually; it gets evalled)

And these ways of storing the last object.

* store in a variable (specify symbol name)
* store in a function cell (specify symbol name)
* store in car of cons cell (specify its sequence number)
* store in cdr of cons cell (specify its sequence number)
* store in array element (specify array sequence number and index)
* store in symbol property (specify symbol sequence number and property name)
* store using expression (specify a lambda expression with one arg, textually)
* store in special internal place (specify a code number to say which place)

And one storage reclaimer

* discard all sequence numbers back to N.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: eliz@gnu.org, npostavs@users.sourceforge.net, emacs-devel@gnu.org
> Date: Thu, 27 Oct 2016 16:39:59 -0400
> 
>   > Eli has mentioned a simpler approach, where we build an .elc file when 
>   > saving Emacs state and load the .elc file during normal startup. The 
>   > main worry about this approach is performance.
> 
> There is no reason why we have to choose between C code and Lisp code.
> It's worth developing a special-purpose format for this, if that would
> be considerably faster.

The Lisp approach has a huge advantage: it is much simpler, so
everyone here will understand it, and it is much easier to maintain
and develop.

So if the performance hit is bearable (meaning will be accepted by the
crowd), it should IMO be preferred for reasons of project management
and its future, even though faster methods exist.  IOW, the goal of
bringing the unexec out of the shadows of system-level black magic it
is now should stomp the "faster is always better" principle, if we
care about the future of Emacs in the face of the fact that fewer and
fewer people know, or even want to know, about segments and offsets in
a binary executable file.

And speaking about performance: I suggest people who worry about that
start by comparing startup times of past versions of Emacs.  Using
this simple benchmark proposed by Andreas:

  time emacs -batch --eval t

I see that we've been consistently adding 10% of startup time with
each major release, beginning with Emacs 23, so 25.1 starts about 25%
slower than 22.3.  If that didn't cause an outcry, then perhaps our
concern for this order of magnitude of differences in startup time are
a tad exaggerated?

Re: When should ralloc.c be used? (WAS: bug#24358)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The Lisp approach has a huge advantage: it is much simpler, so
  > everyone here will understand it, and it is much easier to maintain
  > and develop.

The special format I propose is simple enough.

  > So if the performance hit is bearable (meaning will be accepted by the
  > crowd), it should IMO be preferred for reasons of project management
  > and its future,

Slowness here affects every user and is quite noticeable.
Don't we already know that Lisp is too slow for this?

It is worth substantial extra effort to speed this up.

  > care about the future of Emacs in the face of the fact that fewer and
  > fewer people know, or even want to know, about segments and offsets in
  > a binary executable file.

That is an argument for replacing unexec with something that saves the
data to reloed, but it is not an argument for using Lisp as the format.

  >   Using
  > this simple benchmark proposed by Andreas:

  >   time emacs -batch --eval t

I just tried it with my current build (from June).  It took .26 seconds,
which is fast enough.

If replacing unexec with loading Lisp takes .05 seconds more, I won't
complain.  But I think it will take several seconds, if not minutes.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: eggert@cs.ucla.edu, npostavs@users.sourceforge.net,
> 	emacs-devel@gnu.org
> Date: Fri, 28 Oct 2016 15:12:27 -0400
> 
>   > The Lisp approach has a huge advantage: it is much simpler, so
>   > everyone here will understand it, and it is much easier to maintain
>   > and develop.
> 
> The special format I propose is simple enough.

For you and me (and a few others), maybe.  For most of the current
Emacs contributors it's nowhere near "simple enough", because it
requires one to be familiar with intimate details of Emacs object
design and implementation.

IOW, for the purposes of this discussion, I consider anything that is
not mostly Lisp "not simple".

>   > So if the performance hit is bearable (meaning will be accepted by the
>   > crowd), it should IMO be preferred for reasons of project management
>   > and its future,
> 
> Slowness here affects every user and is quite noticeable.
> Don't we already know that Lisp is too slow for this?

No, we don't know that, because we never tried to implement any method
of reading compiled Lisp that is optimized for speed and targets a
bare Emacs.

E.g., it turned out that most of the time it takes 'loadup' to do its
job is due to the linear search of pure strings in
find_string_data_in_pure, called by make_pure_string.  If we call
'loadup' upon every startup, the need for pure storage goes away, and
the 'loadup' time can be sped up tenfold.  And that is even before
making all of the preloaded files a single file, which speeds up
things at least twofold more, according to my measurements.

So here you have a 20-fold speedup just by two very simple measures.

>   > care about the future of Emacs in the face of the fact that fewer and
>   > fewer people know, or even want to know, about segments and offsets in
>   > a binary executable file.
> 
> That is an argument for replacing unexec with something that saves the
> data to reloed, but it is not an argument for using Lisp as the format.

It is an argument for both, because I don't think we can count on too
many people here being able to tinker with Lisp object internals in
the future.  The less such features we have that will need
maintenance, the better for Emacs viability in the long run.

>   >   time emacs -batch --eval t
> 
> I just tried it with my current build (from June).  It took .26 seconds,
> which is fast enough.
> 
> If replacing unexec with loading Lisp takes .05 seconds more, I won't
> complain.  But I think it will take several seconds, if not minutes.

How much does it take on your system to do this:

  time src/temacs -batch -l loadup

And if you modify Emacs with the patch posted here:

  http://lists.gnu.org/archive/html/emacs-devel/2016-01/msg01049.html

how long does it take temacs to loadup then?

Re: When should ralloc.c be used?

[Stefan Monnier]

> E.g., it turned out that most of the time it takes 'loadup' to do its
> job is due to the linear search of pure strings in
> find_string_data_in_pure, called by make_pure_string.

Indeed.  hash-consing pure objects takes another very significant
percentage of the time.

> If we call 'loadup' upon every startup, the need for pure storage goes
> away, and the 'loadup' time can be sped up tenfold.

Actually, there's no *need* for pure storage in either case.

There are benefits to the use of pure storage, and some of them remain
even if we don't dump: one of the benefits that remains is to reduce the
size of the GC'd heap and hence speed up the GC.  Whether that's
significant enough to bother with it is of course debatable.  But note
also that we could keep the use of pure-space without paying the hefty
price of find_string_data_in_pure (or hash-consing), since these merely
try to make the purespace more compact.  The extra cost is OK when we
dump the result, but it's not worth the trouble if we don't dump.

> And that is even before making all of the preloaded files a single
> file, which speeds up things at least twofold more, according to
> my measurements.

BTW, how large is that single file (I'm curious if its size is
significantly different from the 3.2MB I got for my dumped.elc)?


        Stefan

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sat, 29 Oct 2016 10:55:20 -0400
> 
> BTW, how large is that single file (I'm curious if its size is
> significantly different from the 3.2MB I got for my dumped.elc)?

Its size is 4.2MB.  It's basically a concatenation of all the
preloaded *.elc files, with all but a single preamble removed.

Re: When should ralloc.c be used?

[Stefan Monnier]

>> BTW, how large is that single file (I'm curious if its size is
>> significantly different from the 3.2MB I got for my dumped.elc)?
> Its size is 4.2MB.  It's basically a concatenation of all the
> preloaded *.elc files, with all but a single preamble removed.

OK, so it's pretty much exactly the same size as what I get (the 3.2MB
I get turns into 4.2MB if I print each function/var separately, thus
preventing sharing between them, which is what happens in the normal
.elc files).

Good to know that there's no significant difference between the
two in this regard, thanks.


        Stefan

Re: When should ralloc.c be used? (WAS: bug#24358)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > For you and me (and a few others), maybe.  For most of the current
  > Emacs contributors it's nowhere near "simple enough", because it
  > requires one to be familiar with intimate details of Emacs object
  > design and implementation.

No it doesn't.  The code to look at objects and output them this way
wouldn't have to know any more about how they are represented
than the code for Fprinc.  It would operate using the usual macros
for decomposing objects.

The idea that all C code should be regarded as unmaintainable
is a nonstarter.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: eggert@cs.ucla.edu, npostavs@users.sourceforge.net,
> 	emacs-devel@gnu.org
> Date: Sat, 29 Oct 2016 12:38:48 -0400
> 
>   > For you and me (and a few others), maybe.  For most of the current
>   > Emacs contributors it's nowhere near "simple enough", because it
>   > requires one to be familiar with intimate details of Emacs object
>   > design and implementation.
> 
> No it doesn't.  The code to look at objects and output them this way
> wouldn't have to know any more about how they are represented
> than the code for Fprinc.

The code like in princ (actually in its subroutines) is exactly what I
think we should try to avoid.

> The idea that all C code should be regarded as unmaintainable
> is a nonstarter.

I didn't say it will be unmaintainable, I said its maintenance will be
harder than of Lisp code.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I didn't say it will be unmaintainable, I said its maintenance will be
  > harder than of Lisp code.

That's no horrible thing.  Speeding up startup is important.
If some C code is an effective way to do it, we shouldn't
reject that just because of a general preference for Lisp code.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: eggert@cs.ucla.edu, emacs-devel@gnu.org,
> 	npostavs@users.sourceforge.net
> Date: Mon, 31 Oct 2016 15:18:38 -0400
> 
>   > I didn't say it will be unmaintainable, I said its maintenance will be
>   > harder than of Lisp code.
> 
> That's no horrible thing.

"Horrible" is in the eyes of the beholder.  I think keeping Emacs as
maintainable as possible is very important for its future.

> If some C code is an effective way to do it, we shouldn't
> reject that just because of a general preference for Lisp code.

I'm not rejecting it, just explaining why it shouldn't be the first
priority, IMO.

Re: When should ralloc.c be used?

[Stefan Monnier]

> There is no reason why we have to choose between C code and Lisp code.
> It's worth developing a special-purpose format for this, if that would
> be considerably faster.

We're still investigating how much time can be gained just by optimizing
lread.c.  But yes, maybe another format (which could also be used for
.elc files, of course) would allow reading faster.


        Stefan

Re: When should ralloc.c be used? (WAS: bug#24358)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Eli has mentioned a simpler approach, where we build an .elc file when 
  > saving Emacs state and load the .elc file during normal startup. The 
  > main worry about this approach is performance.

Any such scheme has this problem:
how to find all the places where initialization has stored some sort
of value?  They do not all have ways to access them and set them from Lisp.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Paul Eggert]

On 10/27/2016 01:40 PM, Richard Stallman wrote:
> Any such scheme has this problem:
> how to find all the places where initialization has stored some sort
> of value?  They do not all have ways to access them and set them from Lisp.

My impression is that most such initializations are so small and fast 
that we needn't worry about saving and restoring their state. We can 
simply redo the initialization when Emacs starts up again - this will be 
the default behavior if we leave the temacs initialization code alone, 
which means we'd get this for very little maintenance effort. Any 
counterexamples we can handle specially, by saving and restoring their 
state by hand (so to speak).

Something along the lines of your idea for storage creation should work, 
though we'll have to be careful about destructive operations like setcar 
that can cause an object with an earlier sequence number to point to an 
object with a later sequence number. It's not clear whether it has 
significant advantages over the C-based approach.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Any such scheme has this problem:
  > > how to find all the places where initialization has stored some sort
  > > of value?  They do not all have ways to access them and set them from Lisp.

  > My impression is that most such initializations are so small and fast 
  > that we needn't worry about saving and restoring their state. We can 
  > simply redo the initialization when Emacs starts up again - this will be 
  > the default behavior if we leave the temacs initialization code alone, 
  > which means we'd get this for very little maintenance effort. 

It could be so, but someone will have to try it.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Thu, 27 Oct 2016 22:40:01 -0400
> Cc: emacs-devel@gnu.org
> 
>   > My impression is that most such initializations are so small and fast 
>   > that we needn't worry about saving and restoring their state. We can 
>   > simply redo the initialization when Emacs starts up again - this will be 
>   > the default behavior if we leave the temacs initialization code alone, 
>   > which means we'd get this for very little maintenance effort. 
> 
> It could be so, but someone will have to try it.

We already have a CANNOT_DUMP configuration, which does precisely
that, used by some systems, so this code is in reasonably good shape.
It's just a question of make it fast enough.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Something along the lines of your idea for storage creation should work, 
  > though we'll have to be careful about destructive operations like setcar 
  > that can cause an object with an earlier sequence number to point to an 
  > object with a later sequence number.

Why so?

The purpose of sequence numbers is simply so you can refer to the
objects already made -- not for proving some theorems of
well-foundedness.  Cycles should not be a problem.

  >  It's not clear whether it has 
  > significant advantages over the C-based approach.

Here are two:

* You don't need a C compiler to dump Emacs.

* You don't need to relink to dump Emacs.



-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: eliz@gnu.org, npostavs@users.sourceforge.net, emacs-devel@gnu.org
> Date: Thu, 27 Oct 2016 16:40:14 -0400
> 
>   > Eli has mentioned a simpler approach, where we build an .elc file when 
>   > saving Emacs state and load the .elc file during normal startup. The 
>   > main worry about this approach is performance.
> 
> Any such scheme has this problem:
> how to find all the places where initialization has stored some sort
> of value?  They do not all have ways to access them and set them from Lisp.

The part that must be done in C, like DEFUN etc. will be done at
startup of the Emacs session.  The part that stores values in
variables exposed to Lisp will be either moved to startup.el, or done
at startup in C.  A few variables whose value depends on the build
directory and other stuff that is only known at build time will be
recorded in a special Lisp file created by the build and loaded at
startup as part of the .elc file described above.

Btw, Paul's description about "saving state" in a .elc file is
inaccurate: most of that file is just the preloaded Lisp packages,
like simple.el, subr.el, etc.  There's very little of saved state
there, because that state will simple be created at startup, each time
Emacs starts.

In source terms, most of the "if (!initialized)" parts will be run
each time Emacs starts.

Does that answer your question?

Re: When should ralloc.c be used? (WAS: bug#24358)

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: npostavs@users.sourceforge.net, eggert@cs.ucla.edu,
> 	emacs-devel@gnu.org
> Date: Tue, 25 Oct 2016 19:49:33 -0400
> 
>   >   I think once you said
>   > here:
> 
>   >   http://lists.gnu.org/archive/html/emacs-devel/2016-01/msg01633.html
> 
>   > that you favored replacing unexec by a more portable scheme, there was
>   > no longer any reasons to make our argument stronger.
> 
> In general, I'm in favor of a more portable method.  But we don't have
> one now.  Is it feasible to do?  Is anyone working on one?  

We are warming up.  There are a few ideas, but I'm not sure we have
decided which one is the best yet.

> If not, then I hope we can design, with the Glibc developers, a
> different set of hooks to allow us to make unexec work.

Frankly, I think that ship has sailed, and cannot be turned around.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > If not, then I hope we can design, with the Glibc developers, a
  > > different set of hooks to allow us to make unexec work.

  > Frankly, I think that ship has sailed, and cannot be turned around.

I think that is an exaggeration.  They got rid of ONE set of hooks for
specific practical reasons.  Maybe we can design a different set of
hooks which do the job and which are not a problem for them to
support.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used?

[Stefan Monnier]

> I think that is an exaggeration.  They got rid of ONE set of hooks for
> specific practical reasons.  Maybe we can design a different set of
> hooks which do the job and which are not a problem for them to
> support.

While that is true, I think there is very little motivation to go down
that road even in Emacs's side: this glibc-malloc "issue" is just one
more nail in the unexec coffin, so even if we can find a way back we'd
still be stuck with the problem of doing unexec with address
randomization (for example), and maintenance of unexec (which has proved
less problematic than I expected, over the years, admittedly, but
remains a source of worry).


        Stefan

Re: When should ralloc.c be used?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I think that is an exaggeration.  They got rid of ONE set of hooks for
  > > specific practical reasons.  Maybe we can design a different set of
  > > hooks which do the job and which are not a problem for them to
  > > support.

  > While that is true, I think there is very little motivation to go down
  > that road even in Emacs's side: this glibc-malloc "issue" is just one
  > more nail in the unexec coffin, so even if we can find a way back we'd
  > still be stuck with the problem of doing unexec with address
  > randomization (for example), and maintenance of unexec (which has proved

I just did  'time temacs -batch -l loadup'.  It took over 18 seconds.
We have a long way to go to make that fast enough.

Perhaps what we need is to dump that data verbatim in a format
chosen by us, then relocate all the pointers if necessary.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used?

[Stefan Monnier]

> I just did  'time temacs -batch -l loadup'.  It took over 18 seconds.
> We have a long way to go to make that fast enough.

It's easy to bring it down to 1s: the current loadup.el spend most of
its time in things that are not terribly important (to try and scrape a few
bytes here and there: worthwhile in the context of a one-off dump step,
but not that important otherwise).


        Stefan

Re: When should ralloc.c be used?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > It's easy to bring it down to 1s: the current loadup.el spend most of
  > its time in things that are not terribly important (to try and scrape a few
  > bytes here and there: worthwhile in the context of a one-off dump step,
  > but not that important otherwise).

Please show us.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Fri, 28 Oct 2016 15:13:19 -0400
> Cc: emacs-devel@gnu.org
> 
> I just did  'time temacs -batch -l loadup'.  It took over 18 seconds.
> We have a long way to go to make that fast enough.

Try the patch I pointed to in a previous message, and see what kind of
speedup is possible by 2 simple measures.

Re: When should ralloc.c be used?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I just did  'time temacs -batch -l loadup'.  It took over 18 seconds.
  > > We have a long way to go to make that fast enough.

  > Try the patch I pointed to in a previous message, and see what kind of
  > speedup is possible by 2 simple measures.

It would be a lot of work for me to try that myself, and I think it is
not necessary if you already tried it.  What fractional speedup did
you observe when you tried it?

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Sat, 29 Oct 2016 12:37:34 -0400
> 
>   > Try the patch I pointed to in a previous message, and see what kind of
>   > speedup is possible by 2 simple measures.
> 
> It would be a lot of work for me to try that myself, and I think it is
> not necessary if you already tried it.  What fractional speedup did
> you observe when you tried it?

About 20, as I wrote.

Re: When should ralloc.c be used?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > It would be a lot of work for me to try that myself, and I think it is
  > > not necessary if you already tried it.  What fractional speedup did
  > > you observe when you tried it?

  > About 20, as I wrote.

Sorry, I am not sure what 20 means here.
Was it a speedup of 20%?
A factor of 20?

If it means 20%, it would be a good step, but much more would
be required to make non-dumping ok for starting Emacs.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used?

[Alp Aker]

[-- Attachment #1: Type: text/plain, Size: 680 bytes --]

On Sun, Oct 30, 2016 at 7:33 AM, Richard Stallman <rms@gnu.org> wrote:

> Sorry, I am not sure what 20 means here.
> Was it a speedup of 20%?
> A factor of 20?

He meant a factor of 20.  Here's the original comment:

> E.g., it turned out that most of the time it takes 'loadup' to do its
> job is due to the linear search of pure strings in
> find_string_data_in_pure, called by make_pure_string.  If we call
> 'loadup' upon every startup, the need for pure storage goes away, and
> the 'loadup' time can be sped up tenfold.  And that is even before
> making all of the preloaded files a single file, which speeds up
> things at least twofold more, according to my measurements.

[-- Attachment #2: Type: text/html, Size: 984 bytes --]

Re: When should ralloc.c be used?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Sorry, I am not sure what 20 means here.
  > > Was it a speedup of 20%?
  > > A factor of 20?

  > He meant a factor of 20.  Here's the original comment:

With a 20-times speeduo, maybe it is fast enough.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Sun, 30 Oct 2016 07:33:12 -0400
> 
>   > > It would be a lot of work for me to try that myself, and I think it is
>   > > not necessary if you already tried it.  What fractional speedup did
>   > > you observe when you tried it?
> 
>   > About 20, as I wrote.
> 
> Sorry, I am not sure what 20 means here.
> Was it a speedup of 20%?
> A factor of 20?

A factor of 20.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   Based on what
  > we've learned the hard way during the last couple of weeks, I'd say
  > that all the Emacs versions before 25.2 (including 25.1) will be
  > unstable on such GNU systems to the degree of making them almost
  > unusable.

We should withdraw 25.1, I think.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I think the way to fix those is by systematically looking at the
  > > source for them, rather than by debugging.

  > Yes, but finding out whether this is so is not easy, because the
  > malloc call is sometimes buried very deep.

There are programs that determine call trees.  We could find these
problems by analyzing the output.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: eggert@cs.ucla.edu, emacs-devel@gnu.org,
> 	npostavs@users.sourceforge.net
> Date: Mon, 24 Oct 2016 22:35:55 -0400
> 
>   > > I think the way to fix those is by systematically looking at the
>   > > source for them, rather than by debugging.
> 
>   > Yes, but finding out whether this is so is not easy, because the
>   > malloc call is sometimes buried very deep.
> 
> There are programs that determine call trees.  We could find these
> problems by analyzing the output.

Yes, but the real problem is to determine whether the code needs any
changes at all.  For that, one must understand the control flow, and
figure out whether pointers to buffer text are used across malloc
calls without any updates.  This is the hardest part, because pointers
are frequently passed down to subroutines and to their subroutines,
which use them or call malloc only under certain conditions.  For
example, it could be that a subroutine only calls malloc if the
passed-in pointer does not originate from a buffer object.

This analysis is what makes the source study hard.

Anyway, I think I just finished hunting and fixing those cases, so the
only remaining issue is with regex.c functions, for which we have a
patch that will most probably do the job.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Yes, but the real problem is to determine whether the code needs any
  > changes at all.  For that, one must understand the control flow, and
  > figure out whether pointers to buffer text are used across malloc
  > calls without any updates.

That is true.  But this can find the functions that need to be checked,
because they make buffer pointers and they indirectly call malloc.


-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Perry E. Metzger]

On Mon, 24 Oct 2016 20:09:32 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> >   > It imposes hard-to-fulfill requirements on functions that get
> >   > C pointers to buffer text or to Lisp string data: those
> >   > functions must never call malloc, directly or indirectly.  
> > 
> > I think the way to fix those is by systematically looking at the
> > source for them, rather than by debugging.  
> 
> Yes, but finding out whether this is so is not easy, because the
> malloc call is sometimes buried very deep.

Could this be found by doing a debugging build where malloc
aborts in the conditions where it can't be called directly or
indirectly? Then one could just run that way and find the instances
pretty easily.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: When should ralloc.c be used? (WAS: bug#24358)

[Eli Zaretskii]

> Date: Tue, 25 Oct 2016 19:00:45 -0400
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: rms@gnu.org, npostavs@users.sourceforge.net, eggert@cs.ucla.edu,
>  emacs-devel@gnu.org
> 
> On Mon, 24 Oct 2016 20:09:32 +0300 Eli Zaretskii <eliz@gnu.org> wrote:
> > >   > It imposes hard-to-fulfill requirements on functions that get
> > >   > C pointers to buffer text or to Lisp string data: those
> > >   > functions must never call malloc, directly or indirectly.  
> > > 
> > > I think the way to fix those is by systematically looking at the
> > > source for them, rather than by debugging.  
> > 
> > Yes, but finding out whether this is so is not easy, because the
> > malloc call is sometimes buried very deep.
> 
> Could this be found by doing a debugging build where malloc
> aborts in the conditions where it can't be called directly or
> indirectly?

I don't know how to define those conditions.  If you have concrete
suggestions, please describe them.

Re: When should ralloc.c be used? (WAS: bug#24358)

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > Could this be found by doing a debugging build where malloc
  > aborts in the conditions where it can't be called directly or
  > indirectly?

We could make the functions that create pointers into buffers also
increment a global counter when they do that, and decrement the
counter when done.  malloc would abort if the counter is nonzero.

The hard part would be arranging to reset the counter to zero when
there is a nonlocal exit out of such a region.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

Re: When should ralloc.c be used?

[Stefan Monnier]

>> But that doesn't explain why we'd need to use ralloc in the mean time.
> Why would we not want to use ralloc?  It made a big improvement for
> memory management when I wrote it.

But that was before we were able to use mmap for the allocation of
buffer memory, which is the main source of fragmentation AFAIK.
Also the size of virtual and physical memory was quite different back then,


        Stefan

bug#24358: 25.1.50;

[Sam Halliday]

I've also been experiencing this bug and I raised a thread about it on
the ArchLinux forums:
https://bbs.archlinux.org/viewtopic.php?pid=1662221

I have built the emacs-25 branch with debugging flags "-O0 -gdwarf-4
-g3" as advised.

I can (unreliably) reproduce by typing `M-x ensime` and it seems like
emacs is crashing when loading the .ensime file, which is an
s-expression file that is loaded as data in
https://github.com/ensime/ensime-emacs/blob/master/ensime-config.el#L153-L168
(actually if anybody knows of a more efficient way to load the file,
I'd be keen to update our code, I'm a maintainer).  The exact file
that it is apparently performing the re_search within is
https://gist.github.com/fommil/d906918819cd2632e9864842e1d59b57

I get this on segfault

=====
Thread 1 "emacs" received signal SIGSEGV, Segmentation fault.
0x00000000005bdd06 in re_search_2 (bufp=0xc49b30 <searchbufs+3632>,
str1=0x56e5508 <error: Cannot access memory at address 0x56e5508>,
size1=0,
    str2=0x56e5508 <error: Cannot access memory at address 0x56e5508>,
size2=65565, startpos=646, range=509, regs=0xc4a930 <search_regs>,
stop=1155)
    at regex.c:4464
4464      int len = BYTES_BY_CHAR_HEAD (*p);
=====

and a bt on the core follows (sorry, huge dump).

Could somebody please let me know how to dig into 0x56e5508 (if
relevant)? I have never used gdb in anger.



#0  0x00000000005bdd06 in re_search_2 (bufp=0xc49b30
<searchbufs+3632>, str1=0x56e5508 <error: Cannot access memory at
address 0x56e5508>, size1=0, str2=0x56e5508 <error: Cannot access
memory at address 0x56e5508>, size2=65565, startpos=646, range=509,
regs=0xc4a930 <search_regs>, stop=1155) at regex.c:4464
#1  0x00000000005acc42 in search_buffer (string=14269316, pos=624,
pos_byte=624, lim=1156, lim_byte=1156, n=1, RE=1, trt=0,
inverse_trt=0, posix=false)
    at search.c:1265
#2  0x00000000005ac4b8 in search_command (string=14269316, bound=4626,
noerror=44544, count=0, direction=1, RE=1, posix=false) at
search.c:1058
#3  0x00000000005b00df in Fre_search_forward (regexp=14269316,
bound=4626, noerror=44544, count=0) at search.c:2264
#4  0x00000000005eec29 in Ffuncall (nargs=4, args=0x7ffffffefad8) at eval.c:2709
#5  0x0000000000630d52 in exec_byte_code (bytestr=10818516,
vector=10818549, maxdepth=38, args_template=0, nargs=0, args=0x0) at
bytecode.c:880
#6  0x00000000005ef6f1 in funcall_lambda (fun=10818469, nargs=3,
arg_vector=0xa513f5 <pure+1329237>) at eval.c:2928
#7  0x00000000005eedfa in Ffuncall (nargs=4, args=0x7fffffff0028) at eval.c:2747
#8  0x0000000000630d52 in exec_byte_code (bytestr=10813244,
vector=10813277, maxdepth=22, args_template=0, nargs=0, args=0x0) at
bytecode.c:880
#9  0x00000000005ef6f1 in funcall_lambda (fun=10813165, nargs=3,
arg_vector=0xa4ff5d <pure+1323965>) at eval.c:2928
#10 0x00000000005eedfa in Ffuncall (nargs=4, args=0x7fffffff0558) at eval.c:2747
#11 0x0000000000630d52 in exec_byte_code (bytestr=10810988,
vector=10811021, maxdepth=18, args_template=0, nargs=0, args=0x0) at
bytecode.c:880
#12 0x00000000005ef6f1 in funcall_lambda (fun=10810877, nargs=2,
arg_vector=0xa4f68d <pure+1321709>) at eval.c:2928
#13 0x00000000005eedfa in Ffuncall (nargs=3, args=0x7fffffff0a90) at eval.c:2747
#14 0x0000000000630d52 in exec_byte_code (bytestr=10832908,
vector=50228949, maxdepth=42, args_template=1030, nargs=1,
args=0x7fffffff1190) at bytecode.c:880
#15 0x00000000005ef3b5 in funcall_lambda (fun=50229021, nargs=1,
arg_vector=0x7fffffff1188) at eval.c:2862
#16 0x00000000005eedfa in Ffuncall (nargs=2, args=0x7fffffff1180) at eval.c:2747
#17 0x00000000005ee202 in run_hook_wrapped_funcall (nargs=2,
args=0x7fffffff1180) at eval.c:2433
#18 0x00000000005ee459 in run_hook_with_args (nargs=2,
args=0x7fffffff1180, funcall=0x5ee1b6 <run_hook_wrapped_funcall>) at
eval.c:2514
#19 0x00000000005ee251 in Frun_hook_wrapped (nargs=2,
args=0x7fffffff1180) at eval.c:2448
#20 0x00000000005eea79 in Ffuncall (nargs=3, args=0x7fffffff1178) at eval.c:2678
#21 0x0000000000630d52 in exec_byte_code (bytestr=10832780,
vector=10832813, maxdepth=78, args_template=2058, nargs=2,
args=0x7fffffff1730) at bytecode.c:880
#22 0x00000000005ef3b5 in funcall_lambda (fun=10832733, nargs=2,
arg_vector=0x7fffffff1720) at eval.c:2862
#23 0x00000000005eedfa in Ffuncall (nargs=3, args=0x7fffffff1718) at eval.c:2747
#24 0x0000000000630d52 in exec_byte_code (bytestr=10833076,
vector=10833109, maxdepth=110, args_template=2050, nargs=2,
args=0x7fffffff1ce8)
    at bytecode.c:880
#25 0x00000000005ef3b5 in funcall_lambda (fun=10833029, nargs=2,
arg_vector=0x7fffffff1cd8) at eval.c:2862
#26 0x00000000005eedfa in Ffuncall (nargs=3, args=0x7fffffff1cd0) at eval.c:2747
#27 0x0000000000630d52 in exec_byte_code (bytestr=10832452,
vector=10832485, maxdepth=50, args_template=1030, nargs=1,
args=0x7fffffff22a0) at bytecode.c:880
#28 0x00000000005ef3b5 in funcall_lambda (fun=10832405, nargs=1,
arg_vector=0x7fffffff2298) at eval.c:2862
#29 0x00000000005eedfa in Ffuncall (nargs=2, args=0x7fffffff2290) at eval.c:2747
#30 0x00000000005ebc6f in internal_condition_case_n (bfun=0x5ee7f9
<Ffuncall>, nargs=2, args=0x7fffffff2290, handlers=44544, hfun=
    0x43cbff <safe_eval_handler>) at eval.c:1394
#31 0x000000000043ce50 in safe__call (inhibit_quit=false, nargs=2,
func=9481216, ap=0x7fffffff2350) at xdisp.c:2558
#32 0x000000000043cf43 in safe_call (nargs=2, func=9481216) at xdisp.c:2574
#33 0x000000000043cf7d in safe_call1 (fn=9481216, arg=2498) at xdisp.c:2585
#34 0x000000000043fecc in handle_fontified_prop (it=0x7fffffff3af0) at
xdisp.c:3805
#35 0x000000000043eec7 in handle_stop (it=0x7fffffff3af0) at xdisp.c:3371
#36 0x000000000044bd29 in next_element_from_buffer (it=0x7fffffff3af0)
at xdisp.c:8322
#37 0x0000000000448302 in get_next_display_element (it=0x7fffffff3af0)
at xdisp.c:6922
#38 0x000000000046ec93 in display_line (it=0x7fffffff3af0) at xdisp.c:20598
#39 0x000000000046450e in try_window (window=20368437, pos=...,
flags=1) at xdisp.c:17247
#40 0x0000000000461937 in redisplay_window (window=20368437,
just_this_one_p=false) at xdisp.c:16696
#41 0x000000000045a803 in redisplay_window_0 (window=20368437) at xdisp.c:14487
#42 0x00000000005ebb1d in internal_condition_case_1 (bfun=0x45a7c1
<redisplay_window_0>, arg=20368437, handlers=13659203, hfun=0x45a789
<redisplay_window_error>) at eval.c:1338
#43 0x000000000045a766 in redisplay_windows (window=20368437) at xdisp.c:14467
#44 0x0000000000459893 in redisplay_internal () at xdisp.c:14027
#45 0x000000000045a183 in redisplay_preserve_echo_area (from_where=13)
at xdisp.c:14320
#46 0x0000000000635b28 in Fdelete_process (process=50226429) at process.c:873
#47 0x00000000005eeb78 in Ffuncall (nargs=2, args=0x7fffffff8b48) at eval.c:2698
#48 0x0000000000630d52 in exec_byte_code (bytestr=16680580,
vector=19382253, maxdepth=18, args_template=2058, nargs=2,
args=0x7fffffff90a0) at bytecode.c:880
#49 0x00000000005ef3b5 in funcall_lambda (fun=48360773, nargs=2,
arg_vector=0x7fffffff9090) at eval.c:2862
#50 0x00000000005eedfa in Ffuncall (nargs=3, args=0x7fffffff9088) at eval.c:2747
#51 0x0000000000630d52 in exec_byte_code (bytestr=54486340,
vector=47296317, maxdepth=30, args_template=1030, nargs=1,
args=0x7fffffff95b8) at bytecode.c:880
#52 0x00000000005ef3b5 in funcall_lambda (fun=47296397, nargs=1,
arg_vector=0x7fffffff95b0) at eval.c:2862
#53 0x00000000005eedfa in Ffuncall (nargs=2, args=0x7fffffff95a8) at eval.c:2747
#54 0x0000000000630d52 in exec_byte_code (bytestr=54504036,
vector=47287597, maxdepth=10, args_template=2, nargs=0,
args=0x7fffffff9ad0) at bytecode.c:880
#55 0x00000000005ef3b5 in funcall_lambda (fun=47287653, nargs=0,
arg_vector=0x7fffffff9ad0) at eval.c:2862
#56 0x00000000005eedfa in Ffuncall (nargs=1, args=0x7fffffff9ac8) at eval.c:2747
#57 0x0000000000630d52 in exec_byte_code (bytestr=54527172,
vector=47283725, maxdepth=6, args_template=2, nargs=0,
args=0x7fffffffa080) at bytecode.c:880
#58 0x00000000005ef3b5 in funcall_lambda (fun=47283773, nargs=0,
arg_vector=0x7fffffffa080) at eval.c:2862
#59 0x00000000005eedfa in Ffuncall (nargs=1, args=0x7fffffffa078) at eval.c:2747
#60 0x00000000005ee058 in funcall_nil (nargs=1, args=0x7fffffffa078)
at eval.c:2337
#61 0x00000000005ee459 in run_hook_with_args (nargs=1,
args=0x7fffffffa078, funcall=0x5ee035 <funcall_nil>) at eval.c:2514
#62 0x00000000005ee0df in Frun_hook_with_args (nargs=1,
args=0x7fffffffa078) at eval.c:2379
#63 0x00000000005ee4b8 in run_hook (hook=41424256) at eval.c:2527
#64 0x000000000057a033 in Fkill_buffer (buffer_or_name=37882885) at
buffer.c:1680
#65 0x00000000005eeb78 in Ffuncall (nargs=2, args=0x7fffffffa228) at eval.c:2698
#66 0x0000000000630d52 in exec_byte_code (bytestr=52539684,
vector=29921029, maxdepth=22, args_template=0, nargs=0, args=0x0) at
bytecode.c:880
#67 0x00000000005ef6f1 in funcall_lambda (fun=29921173, nargs=1,
arg_vector=0x1c88f05) at eval.c:2928
#68 0x00000000005eedfa in Ffuncall (nargs=2, args=0x7fffffffa758) at eval.c:2747
#69 0x0000000000630d52 in exec_byte_code (bytestr=41688580,
vector=54799629, maxdepth=34, args_template=0, nargs=0, args=0x0) at
bytecode.c:880
#70 0x00000000005ef6f1 in funcall_lambda (fun=54799973, nargs=1,
arg_vector=0x3442d0d) at eval.c:2928
#71 0x00000000005eedfa in Ffuncall (nargs=2, args=0x7fffffffaca8) at eval.c:2747
#72 0x0000000000630d52 in exec_byte_code (bytestr=41686276,
vector=54799229, maxdepth=26, args_template=0, nargs=0, args=0x0) at
bytecode.c:880
#73 0x00000000005ef6f1 in funcall_lambda (fun=54799437, nargs=1,
arg_vector=0x3442b7d) at eval.c:2928
#74 0x00000000005eedfa in Ffuncall (nargs=2, args=0x7fffffffb1e8) at eval.c:2747
#75 0x0000000000630d52 in exec_byte_code (bytestr=41687076,
vector=54799477, maxdepth=10, args_template=0, nargs=0, args=0x0) at
bytecode.c:880
#76 0x00000000005ef6f1 in funcall_lambda (fun=54799517, nargs=1,
arg_vector=0x3442c75) at eval.c:2928
#77 0x00000000005ef137 in apply_lambda (fun=54799517, args=43886963,
count=24) at eval.c:2799
#78 0x00000000005ed9d6 in eval_sub (form=43886947) at eval.c:2216
#79 0x00000000005e9efc in Fprogn (body=43886931) at eval.c:431
#80 0x00000000005ef660 in funcall_lambda (fun=43886899, nargs=1,
arg_vector=0x7fffffffb980) at eval.c:2921
#81 0x00000000005eeecb in Ffuncall (nargs=2, args=0x7fffffffb978) at eval.c:2759
#82 0x0000000000630d52 in exec_byte_code (bytestr=22881828,
vector=44171061, maxdepth=22, args_template=0, nargs=0, args=0x0) at
bytecode.c:880
#83 0x00000000005ef6f1 in funcall_lambda (fun=44171189, nargs=3,
arg_vector=0x2a1ff35) at eval.c:2928
#84 0x00000000005eedfa in Ffuncall (nargs=4, args=0x7fffffffbea8) at eval.c:2747
#85 0x0000000000630d52 in exec_byte_code (bytestr=39202884,
vector=58046421, maxdepth=30, args_template=0, nargs=0, args=0x0) at
bytecode.c:880
#86 0x0000000000630153 in Fbyte_code (bytestr=39202884,
vector=58046421, maxdepth=30) at bytecode.c:449
#87 0x00000000005ed82a in eval_sub (form=54684739) at eval.c:2177
#88 0x00000000005eb9f7 in internal_lisp_condition_case (var=9356672,
bodyform=54684739, handlers=54684499) at eval.c:1285
#89 0x0000000000631ee7 in exec_byte_code (bytestr=39203140,
vector=58046541, maxdepth=14, args_template=0, nargs=0, args=0x0) at
bytecode.c:1119
#90 0x00000000005ef6f1 in funcall_lambda (fun=58046597, nargs=0,
arg_vector=0x375b84d) at eval.c:2928
#91 0x00000000005eedfa in Ffuncall (nargs=1, args=0x7fffffffcb68) at eval.c:2747
#92 0x00000000005e7151 in Ffuncall_interactively (nargs=1,
args=0x7fffffffcb68) at callint.c:252
#93 0x00000000005eea79 in Ffuncall (nargs=2, args=0x7fffffffcb60) at eval.c:2678
#94 0x00000000005edca2 in Fapply (nargs=3, args=0x7fffffffcb60) at eval.c:2279
#95 0x00000000005e75c9 in Fcall_interactively (function=1641232,
record_flag=5302080, keys=13460885) at callint.c:389
#96 0x00000000005eebe6 in Ffuncall (nargs=4, args=0x7fffffffcde8) at eval.c:2705
#97 0x0000000000630d52 in exec_byte_code (bytestr=10383612,
vector=10383645, maxdepth=54, args_template=4102, nargs=2,
args=0x7fffffffd368) at bytecode.c:880
#98 0x00000000005ef3b5 in funcall_lambda (fun=10383565, nargs=2,
arg_vector=0x7fffffffd358) at eval.c:2862
#99 0x00000000005eedfa in Ffuncall (nargs=3, args=0x7fffffffd350) at eval.c:2747
#100 0x0000000000630d52 in exec_byte_code (bytestr=10382812,
vector=10382845, maxdepth=62, args_template=3078, nargs=3,
args=0x7fffffffd9a8)
    at bytecode.c:880
#101 0x00000000005ef3b5 in funcall_lambda (fun=10382757, nargs=3,
arg_vector=0x7fffffffd990) at eval.c:2862
#102 0x00000000005eedfa in Ffuncall (nargs=4, args=0x7fffffffd988) at
eval.c:2747
#103 0x00000000005e7151 in Ffuncall_interactively (nargs=4,
args=0x7fffffffd988) at callint.c:252
#104 0x00000000005eea79 in Ffuncall (nargs=5, args=0x7fffffffd980) at
eval.c:2678
#105 0x00000000005edffe in Fapply (nargs=3, args=0x7fffffffda70) at eval.c:2326
#106 0x00000000005e75c9 in Fcall_interactively (function=673296,
record_flag=0, keys=13460885) at callint.c:389
#107 0x00000000005eebe6 in Ffuncall (nargs=4, args=0x7fffffffdcf8) at
eval.c:2705
#108 0x0000000000630d52 in exec_byte_code (bytestr=10383612,
vector=10383645, maxdepth=54, args_template=4102, nargs=1,
args=0x7fffffffe250)
    at bytecode.c:880
#109 0x00000000005ef3b5 in funcall_lambda (fun=10383565, nargs=1,
arg_vector=0x7fffffffe248) at eval.c:2862
#110 0x00000000005eedfa in Ffuncall (nargs=2, args=0x7fffffffe240) at
eval.c:2747
#111 0x00000000005ee5a1 in call1 (fn=14832, arg1=673296) at eval.c:2557
#112 0x000000000055482b in command_loop_1 () at keyboard.c:1484
#113 0x00000000005eba83 in internal_condition_case (bfun=0x554075
<command_loop_1>, handlers=19104, hfun=0x553862 <cmd_error>) at
eval.c:1314
#114 0x0000000000553d7f in command_loop_2 (ignore=0) at keyboard.c:1112
#115 0x00000000005eb3a3 in internal_catch (tag=45936, func=0x553d56
<command_loop_2>, arg=0) at eval.c:1079
#116 0x0000000000553d21 in command_loop () at keyboard.c:1091
#117 0x000000000055342a in recursive_edit_1 () at keyboard.c:697
#118 0x00000000005535be in Frecursive_edit () at keyboard.c:768
#119 0x0000000000551445 in main (argc=1, argv=0x7fffffffe6f8) at emacs.c:1626

bug#24358: 25.1.50;

[Sam Halliday]

I've attempted to workaround this in ensime by simplifying our loading
of the .ensime file (the s-expression) to use with-temp-buffer and not
declaring .ensime as an emacs-lisp file... it may be that one of my
minor modes, active in emacs-lisp, is performing a regexp that is
triggering this bug.

bug#24358: 25.1.50;

[Eli Zaretskii]

> From: Sam Halliday <sam.halliday@gmail.com>
> Date: Tue, 18 Oct 2016 09:16:38 +0100
> 
> Could somebody please let me know how to dig into 0x56e5508 (if
> relevant)? I have never used gdb in anger.

The factor that triggers the bug is the call to malloc, which
causes relocation of buffer text to a different address, while
re_search_2 still uses the (stale) pointer to that text.

So the way to make sure this is the same bug is to see that
(1) there's a call to malloc between entry to re_search_2 and
the crash, and (2) that the value of current_buffer->text->beg
is different before and after the call to malloc.

Thanks.