bug#32503: 26.1; Byte-compiled functions don't hash consistently

bug#32503: 26.1; Byte-compiled functions don't hash consistently

[Adam Porter]

I noticed that byte-compiled functions don't hash consistently.  Here's
an ECM from Noam Postavsky
<https://github.com/skeeto/emacs-memoize/pull/10#issuecomment-415066682>:

    (let ((obj1 (byte-compile (lambda (x) x)))
          (obj2 (byte-compile (lambda (x) x))))
      (list (equal obj1 obj2)
            (eq obj1 obj2)
            (= (sxhash obj1)
               (sxhash obj2))))
    ;=> (t nil nil)

According to "(elisp) Defining Hash":

    If two objects OBJ1 and OBJ2 are equal, then ‘(sxhash OBJ1)’ and
    ‘(sxhash OBJ2)’ are the same integer.

This makes it impossible to use byte-compiled functions as keys or parts
of keys in hash tables.  A workaround is to `prin1-to-string` the
byte-compiled function and use that as the key.

Thanks.

bug#32503: 26.1; Byte-compiled functions don't hash consistently

[Tom Tromey]

Adam> I noticed that byte-compiled functions don't hash consistently.

The Lisp_Vectorlike case in Fequal has a number of special cases which
don't seem to be reflected in sxhash:

	if (OVERLAYP (o1))
[...]
	if (MARKERP (o1))
[...]
	if (WINDOW_CONFIGURATIONP (o1))
[...]
	/* Aside from them, only true vectors, char-tables, compiled
	   functions, and fonts (font-spec, font-entity, font-object)
	   are sensible to compare, so eliminate the others now.  */

(Bignums and bool vectors are special-cased as well but are handled in
sxhash.)  This is followed by code to loop over the pseudovector's
elements.

However, the Lisp_Vectorlike case in sxhash doesn't do any of this.

Probably there should be a rule that adding a new type should
necessitate adding a new hasher (much like how print syntax is needed);
or certainly if the type has non-trivial equal behavior.

Tom

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[NiwTinray via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi, Emacs Dev, 

I was testing the new portable dumper with my personal Emacs. When I
loaded my dumped file, Emacs crashed on segmentation fault.
After tracking down the issue, I found the issues is caused by the
undo-tree mode in package "evil".
To reproduce the bug, try this:

emacs --batch -f package-initialize --eval='(use-package evil :ensure
t)' --eval='(dump-emacs-portable "test.pdmp")'  && emacs --dump-file
test.pdmp

If I turned off the global-undo-tree-mode manually, it won't crash:

emacs --batch -f package-initialize --eval='(use-package evil :ensure
t)' --eval='(global-undo-tree-mode -1)' --eval='(dump-emacs-portable
"test.pdmp")'  && emacs --dump-file test.pdmp

So the problem must be exist in the undo-tree-mode.

I don't have the knowledge/techniques to track down the bug
further. Please help me investigate this!

Backtrace on crash:
#0  0x00000000004f1240 in Fcurrent_active_maps (olp=olp@entry=0x30, position=position@entry=0x0) at keymap.c:1541
#1  0x00000000004f167d in Fkey_binding (key=0x7fffefba745d, accept_default=0x0, no_remap=0x0, position=0x0) at keymap.c:1681
#2  0x00000000005546a3 in Ffuncall (nargs=2, args=args@entry=0x7fffffffd5d0) at eval.c:2794
#3  0x0000000000588d28 in exec_byte_code (bytestr=<optimized out>, vector=0x7fffefba2715, maxdepth=<optimized out>, args_template=<optimized out>, nargs=nargs@entry=0, args=<optimized out>, args@entry=0x7fffefba2718) at bytecode.c:633
#4  0x000000000055439f in funcall_lambda (fun=0x7fffffffd5e3, nargs=nargs@entry=0, arg_vector=0x7fffefba2718, arg_vector@entry=0x7fffffffd768) at eval.c:2989
#5  0x00000000005545ef in Ffuncall (nargs=1, args=args@entry=0x7fffffffd760) at eval.c:2808
#6  0x0000000000588d28 in exec_byte_code (bytestr=<optimized out>, vector=0x7fffefba262d, maxdepth=<optimized out>, args_template=<optimized out>, nargs=nargs@entry=0, args=<optimized out>, args@entry=0x7fffefba2630) at bytecode.c:633
#7  0x000000000055439f in funcall_lambda (fun=0x7fffffffd79e, nargs=nargs@entry=0, arg_vector=0x7fffefba2630, arg_vector@entry=0x7fffffffd958) at eval.c:2989
#8  0x00000000005545ef in Ffuncall (nargs=1, args=0x7fffffffd950) at eval.c:2808
#9  0x00000000005546e9 in funcall_nil (nargs=<optimized out>, args=<optimized out>) at eval.c:2435
#10 0x00000000005534a5 in run_hook_with_args (nargs=1, args=0x7fffffffd950, funcall=0x5546e0 <funcall_nil>) at eval.c:2612
#11 0x0000000000553616 in Frun_hook_with_args (args=0x7fffffffd950, nargs=1) at eval.c:2477
#12 run_hook (hook=0x7fffeefb53f0) at eval.c:2625
#13 Frun_hooks (nargs=1, args=0x7fffffffd9f8) at eval.c:2459
#14 0x00000000005546a3 in Ffuncall (nargs=2, args=args@entry=0x7fffffffd9f0) at eval.c:2794
#15 0x0000000000588d28 in exec_byte_code (bytestr=<optimized out>, vector=0x7fffefab5415, maxdepth=<optimized out>, args_template=<optimized out>, nargs=nargs@entry=1, args=<optimized out>, args@entry=0x7fffefab5418) at bytecode.c:633
#16 0x000000000055439f in funcall_lambda (fun=0x7fffffffda66, nargs=nargs@entry=1, arg_vector=0x7fffefab5418, arg_vector@entry=0x7fffffffdbd0) at eval.c:2989
#17 0x00000000005545ef in Ffuncall (nargs=2, args=args@entry=0x7fffffffdbc8) at eval.c:2808
#18 0x0000000000588d28 in exec_byte_code (bytestr=<optimized out>, vector=0x7fffefc794ad, maxdepth=<optimized out>, args_template=<optimized out>, nargs=nargs@entry=0, args=<optimized out>, args@entry=0x7fffefc794b0) at bytecode.c:633
#19 0x000000000055439f in funcall_lambda (fun=0x7fffffffdbf1, nargs=nargs@entry=0, arg_vector=0x7fffefc794b0, arg_vector@entry=0x7fffffffdd50) at eval.c:2989
#20 0x00000000005545ef in Ffuncall (nargs=nargs@entry=1, args=args@entry=0x7fffffffdd48) at eval.c:2808
#21 0x0000000000554758 in call0 (fn=0x7fffef08c260) at eval.c:2647
#22 0x000000000050a4ff in get_minibuffer (depth=depth@entry=0) at minibuf.c:754
#23 0x0000000000501283 in init_buffer () at buffer.c:5430
#24 0x00000000004139ba in main (argc=3, argv=<optimized out>) at emacs.c:1777






In GNU Emacs 27.0.60 (build 1, x86_64-pc-linux-gnu)
 of 2020-01-01 built on omnisky
Repository revision: 186152ba400b58d2d278c52d2e3d896decae767e
Repository branch: emacs-27
System Description: Ubuntu 16.04.6 LTS

Recent messages:
Saving file /home/ntr/.config/emacs/custom.el...
Wrote /home/ntr/.config/emacs/emacs.saves/.!home!ntr!.config!emacs!custom.el.~undo-tree~
Wrote /home/ntr/.config/emacs/custom.el
helm-M-x: Package ‘evil-20190729.704’ is used by ‘evil-multiedit’ as dependency, not deleting
Mark saved where search started
Package ‘highlight-parentheses-20180704.1102’ deleted.
Mark saved where search started
Saving file /home/ntr/.config/emacs/bunny-core-packages.el...
Wrote /home/ntr/.config/emacs/emacs.saves/.!home!ntr!.config!emacs!bunny-core-packages.el.~undo-tree~
Wrote /home/ntr/.config/emacs/bunny-core-packages.el

Configured using:
 'configure --with-x-toolkit=no'

Configured features:
XPM JPEG TIFF GIF PNG SOUND GSETTINGS GLIB NOTIFY INOTIFY GNUTLS
FREETYPE HARFBUZZ XFT ZLIB OLDXMENU X11 XDBE XIM MODULES THREADS JSON
PDUMPER GMP

Important settings:
  value of $LC_ALL: en_US.UTF-8
  value of $LC_CTYPE: zh_CN.UTF-8
  value of $LC_MONETARY: zh_CN.UTF-8
  value of $LC_NUMERIC: zh_CN.UTF-8
  value of $LC_TIME: zh_CN.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Emacs-Lisp

Minor modes in effect:
  doom-modeline-mode: t
  recentf-mode: t
  xterm-mouse-mode: t
  display-time-mode: t
  workgroups-mode: t
  keyfreq-autosave-mode: t
  keyfreq-mode: t
  global-magit-file-mode: t
  magit-file-mode: t
  magit-auto-revert-mode: t
  global-git-commit-mode: t
  async-bytecomp-package-mode: t
  company-statistics-mode: t
  global-company-mode: t
  company-mode: t
  global-hungry-delete-mode: t
  hungry-delete-mode: t
  eyebrowse-mode: t
  helm-mode: t
  override-global-mode: t
  delete-selection-mode: t
  auto-revert-mode: t
  global-hl-line-mode: t
  which-key-mode: t
  global-evil-matchit-mode: t
  evil-matchit-mode: t
  global-evil-collection-unimpaired-mode: t
  evil-collection-unimpaired-mode: t
  evil-leader-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  shell-dirtrack-mode: t
  evil-mode: t
  evil-local-mode: t
  global-hl-todo-mode: t
  hl-todo-mode: t
  global-highlight-parentheses-mode: t
  highlight-parentheses-mode: t
  aggressive-indent-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(evil-matchit-simple helm-ring em-unix em-term em-script em-prompt em-ls
em-hist em-pred em-dirs esh-var em-cmpl em-basic em-banner em-alias
esh-mode eshell esh-cmd esh-ext esh-opt esh-proc esh-io esh-arg
esh-module esh-groups shadow sort mail-extr emacsbug sendmail tabify ido
face-remap image-file mule-util view lsp-clients lsp-haxe lsp-erlang
lsp-fsharp lsp-metals lsp-elm lsp-dart lsp-clojure lsp-go lsp-xml
lsp-css lsp-intelephense lsp-vetur lsp-html lsp-solargraph lsp-rust
lsp-pyls lsp helm-for-files helm-bookmark helm-adaptive helm-external
helm-net ffap vc-mtn vc-hg vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs vc
vc-dispatcher jka-compr eieio-opt speedbar sb-image ezimage dframe
help-fns radix-tree so-long winner helm-command helm-elisp helm-eval
edebug backtrace helm-info term/xterm xterm doom-modeline
doom-modeline-segments doom-modeline-env doom-modeline-core recentf
magit-bookmark bookmark ediff ediff-merg ediff-mult ediff-wind
ediff-diff ediff-help ediff-init ediff-util xt-mouse bunny-sshfs
flymake-diagnostic-at-point popup lsp-python-ms
python-el-fgallina-expansions python tramp-sh tramp tramp-loaddefs
trampver tramp-integration files-x tramp-compat parse-time iso8601
ls-lisp company-lsp lsp-mode ewoc markdown-mode tree-widget spinner
network-stream nsm inline ht em-glob esh-util dash-functional bindat
flymake-proc flymake warnings bunny-pyenv eldoc-eval all-the-icons
all-the-icons-faces data-material data-weathericons data-octicons
data-fileicons data-faicons data-alltheicons memoize eshell-git-prompt
leuven-theme time bunny-insert-surroundings bunny-krws
bunny-company-simple-complete bunny-eshell-extensions
bunny-terminal-here bunny-workgroups bunny-register-jumper bunny-h5ls
bunny-prettify-json-file web-server web-server-status-codes keyfreq
evil-magit magit-submodule magit-obsolete magit-blame magit-stash
magit-reflog magit-bisect magit-push magit-pull magit-fetch magit-clone
magit-remote magit-commit magit-sequence magit-notes magit-worktree
magit-tag magit-merge magit-branch magit-reset magit-files magit-refs
magit-status magit magit-repos magit-apply magit-wip magit-log
which-func imenu magit-diff smerge-mode magit-core magit-autorevert
magit-margin magit-transient magit-process magit-mode git-commit
magit-git magit-section magit-utils crm log-edit message rmc rfc822 mml
mml-sec epa derived epg epg-config mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader pcvs-util add-log
with-editor async-bytecomp server git-timemachine transient vc-git
diff-mode org-preview-html eww mm-url gnus nnheader gnus-util rmail
rmail-loaddefs rfc2047 rfc2045 ietf-drums mail-utils mm-util mail-prsvr
url-queue shr text-property-search puny svg xml the-org-mode-expansions
org ob ob-tangle ob-ref ob-lob ob-table ob-exp org-macro org-footnote
org-src ob-comint org-pcomplete org-list org-faces org-entities
time-date noutline outline org-version ob-emacs-lisp ob-core ob-eval
org-table ol org-keys org-compat org-macs org-loaddefs find-func
cal-menu calendar cal-loaddefs company-statistics company-oddmuse
company-keywords company-etags etags fileloop generator company-gtags
company-dabbrev-code company-dabbrev company-files company-capf
company-cmake company-xcode company-clang company-semantic company-eclim
company-template company-bbdb company pcase multi-term shell-pop term
disp-table ehelp htmlize hungry-delete expand-region
text-mode-expansions html-mode-expansions er-basic-expansions
expand-region-core expand-region-custom golden-ratio eyebrowse
use-package-diminish buffer-move transpose-frame helm-ag helm-descbinds
helm-projectile cus-edit cus-start cus-load wid-edit helm-mode
helm-files helm-buffers helm-occur helm-tags helm-locate helm-grep
helm-regexp format-spec helm-utils helm-help helm-types helm helm-source
eieio-compat helm-multi-match helm-lib async use-package-bind-key
bind-key counsel xdg xref project swiper ivy delsel colir ivy-overlay
ace-window avy ace-jump-zap ace-jump-mode cl ranger autorevert
filenotify hl-line evil-collection-dired dired dired-loaddefs
evil-collection-neotree neotree projectile grep compile ibuf-ext ibuffer
ibuffer-loaddefs shrink-path rx f s dash which-key evil-multiedit iedit
iedit-lib evil-nerd-commenter evil-nerd-commenter-operator
evil-nerd-commenter-sdk sgml-mode dom evil-matchit evil-matchit-sdk
evil-collection-unimpaired evil-collection
bunny-minor-mode-leader-keymap zone-rainbow color zone symbol-overlay
evil-leader evil evil-integration undo-tree diff evil-maps evil-commands
reveal flyspell ispell evil-jumps evil-command-window evil-types
evil-search evil-ex shell pcomplete comint ansi-color evil-macros
evil-repeat evil-states evil-core advice evil-common windmove thingatpt
rect evil-digraphs evil-vars edmacro kmacro hl-todo posframe
rainbow-delimiters highlight-parentheses move-text macrostep ring pp
exec-path-from-shell try url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-domsuf url-util mailcap anaphora
aggressive-indent easy-mmode cl-extra help-mode use-package-ensure
use-package-core finder-inf info package easymenu browse-url
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json subr-x map url-vars seq byte-opt gv
bytecomp byte-compile cconv cl-loaddefs cl-lib tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads inotify dynamic-setting system-font-setting
font-render-setting x multi-tty make-network-process emacs)

Memory information:
((conses 16 632937 99192)
 (symbols 48 53853 52)
 (strings 32 181171 13075)
 (string-bytes 1 5457906)
 (vectors 16 89834)
 (vector-slots 8 1780900 209238)
 (floats 8 893 401)
 (intervals 56 24129 5564)
 (buffers 1000 48)
 (heap 1024 49509 3144))

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Eli Zaretskii]

> Date: Sat, 4 Jan 2020 09:49:14 +0800
> From: NiwTinray via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> I was testing the new portable dumper with my personal Emacs. When I
> loaded my dumped file, Emacs crashed on segmentation fault.
> After tracking down the issue, I found the issues is caused by the
> undo-tree mode in package "evil".
> To reproduce the bug, try this:
> 
> emacs --batch -f package-initialize --eval='(use-package evil :ensure
> t)' --eval='(dump-emacs-portable "test.pdmp")'  && emacs --dump-file
> test.pdmp

I cannot reproduce this from "emacs -Q" because 'use-package' is not a
known function.  Can you show a recipe starting from "emacs -Q",
please?

Also, does this happen if you add -Q to the Emacs invocation after
dumping?  If not, there's more detail missing in your report: the
customizations in your init files.

In addition, please also show the Lisp-level backtrace from the crash,
by using the xbacktrace command (it is defined in the src/.gdbinit
file in the Emacs source tree).

> Backtrace on crash:
> #0  0x00000000004f1240 in Fcurrent_active_maps (olp=olp@entry=0x30, position=position@entry=0x0) at keymap.c:1541

GDB usually displays the fatal signal that killed the program; can you
show that part as well?

Thanks.

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Eli Zaretskii]

[Please use "Reply All" to reply, so that the bug address is kept on
the CC list.]

> From: NiwTinray <niwtrx@icloud.com>
> Date: Sun, 5 Jan 2020 13:25:07 +0800
> 
> > I cannot reproduce this from "emacs -Q" because 'use-package' is not a
> > known function.  Can you show a recipe starting from "emacs -Q",
> > please?
> 
> Here. I've attached a minimal script file that helps reproduce this bug.
> 
> (require 'package)
> (package-initialize)
> (add-to-list 'package-archives
>              '("melpa-stable" . "https://stable.melpa.org/packages/") t)
> (unless (package-installed-p 'evil)
>   (package-refresh-contents)
>   (package-install 'evil))
> (require 'evil)
> (dump-emacs-portable "/tmp/test.pdmp")
> 
> The script downloads the package "evil" from Melpa stable, load the evil package
> and dumps an image to /tmp/test.pdmp.
> 
> > Also, does this happen if you add -Q to the Emacs invocation after
> > dumping?  If not, there's more detail missing in your report: the
> > customizations in your init files.
> 
> 
> Sure. Please download this file, and run the command: 
> 
> emacs --batch -Q --script evil.el
> 
> To see the bug happen, load the test.pdmp file: 
> 
> emacs -Q --dump-file /tmp/test.pdmp
> 
> You should see a segmentation fault:
> 
> [1]    23369 segmentation fault (core dumped)  emacs -Q --dump-file /tmp/test.pdmp
> 
> I run debugger inside src/.gdbinit using the command:
> 
> gdb -x .gdbinit --args ./emacs --dump-file /tmp/test.pdmp
> 
> And logged backtrace. See my second attachment: 
> 
> (base) omnisky :: ~/emacs/src ‹emacs-27*› » gdb -x .gdbinit --args ./emacs --dump-file /tmp/test.pdmp
> GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
> Copyright (C) 2016 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from ./emacs...done.
> warning: File "/home/ntr/emacs/src/.gdbinit" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
> To enable execution of this file add
> 	add-auto-load-safe-path /home/ntr/emacs/src/.gdbinit
> line to your configuration file "/home/ntr/.gdbinit".
> To completely disable this security protection add
> 	set auto-load safe-path /
> line to your configuration file "/home/ntr/.gdbinit".
> For more information about this security protection see the
> "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
> 	info "(gdb)Auto-loading safe path"
> SIGINT is used by the debugger.
> Are you sure you want to change it? (y or n) [answered Y; input not from terminal]
> Environment variable "DISPLAY" not defined.
> TERM = xterm-24bits
> Breakpoint 1 at 0x411df0: file emacs.c, line 370.
> Breakpoint 2 at 0x4bfe60: file xterm.c, line 10130.
> (gdb) r
> Starting program: /home/ntr/emacs/src/emacs --dump-file /tmp/test.pdmp
> /home/ntr/emacs/src/emacs: /raid_sdc/home/ntr/anaconda3/lib/libtiff.so.5: no version information available (required by /home/ntr/emacs/src/emacs)
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> 
> Program received signal SIGSEGV, Segmentation fault.
> 0x00000000004f12d0 in Fcurrent_active_maps (olp=olp@entry=XIL(0x30), position=position@entry=XIL(0)) at keymap.c:1541
> 1541	      && NILP (KVAR (current_kboard, Voverriding_terminal_local_map))
> (gdb) xbacktrace
> "key-binding" (0xffffd5c8)
> "turn-on-undo-tree-mode" (0xffffd758)
> "global-undo-tree-mode-enable-in-buffers" (0xffffd948)
> "run-hooks" (0xffffd9e8)
> "run-mode-hooks" (0xffffdbc0)
> "minibuffer-inactive-mode" (0xffffdd40)
> (gdb)

In my debug build of Emacs 27.0.60 I get an assertion violation while
dumping, which probably is already a sign of trouble.

Daniel, any ideas?

Here's the backtrace from the assertion violation, and some data
involved in the assertion:

  dumping fingerprint: 79862409ba15bcbb091a8b1aa5b942cc3283f12f123a69372f5cfe59de047ba9

  pdumper.c:2684: Emacs fatal error: assertion failed: EQ (expected_value, found_value)

  Thread 1 hit Breakpoint 1, terminate_due_to_signal (sig=22,
      backtrace_limit=2147483647) at emacs.c:371
  371       signal (sig, SIG_DFL);
  (gdb) bt
  #0  terminate_due_to_signal (sig=22, backtrace_limit=2147483647)
      at emacs.c:371
  #1  0x01317f22 in die (
      msg=0x19bac5c <WEIGHT_STRONG+2544> "EQ (expected_value, found_value)",
      file=0x19ba210 <dump_magic+16> "pdumper.c", line=2684) at alloc.c:7246
  #2  0x01326817 in check_hash_table_rehash (table_orig=XIL(0xa000000006288090))
      at pdumper.c:2684
  #3  0x01326a6c in dump_hash_table (ctx=0x82d1f0,
      object=XIL(0xa000000006288090), offset=-4) at pdumper.c:2725
  #4  0x013279c5 in dump_vectorlike (ctx=0x82d1f0, lv=XIL(0xa000000006288090),
      offset=-4) at pdumper.c:2991
  #5  0x01327f92 in dump_object (ctx=0x82d1f0, object=XIL(0xa000000006288090))
      at pdumper.c:3127
  #6  0x0132ace3 in dump_drain_deferred_hash_tables (ctx=0x82d1f0)
      at pdumper.c:3977
  #7  0x0132b4ca in Fdump_emacs_portable (filename=XIL(0x8000000006d24008),
      track_referrers=XIL(0)) at pdumper.c:4148
  #8  0x013825de in eval_sub (form=XIL(0xc0000000067b2150)) at eval.c:2276
  #9  0x0137aec2 in Fprogn (body=XIL(0)) at eval.c:462
  #10 0x01382182 in eval_sub (form=XIL(0xc0000000067b2090)) at eval.c:2226
  #11 0x013819e8 in Feval (form=XIL(0xc0000000067b2090), lexical=XIL(0x30))
      at eval.c:2102
  #12 0x01384ecc in funcall_subr (subr=0x1960b00 <Seval>, numargs=2,
      args=0x82d978) at eval.c:2869
  #13 0x013848f9 in Ffuncall (nargs=3, args=0x82d970) at eval.c:2794
  #14 0x01427e37 in exec_byte_code (bytestr=XIL(0x800000000612e1e0),
      vector=XIL(0xa00000000612d0f8), maxdepth=make_fixnum(25),
      args_template=make_fixnum(257), nargs=1, args=0x82e378) at bytecode.c:633
  #15 0x01385a11 in funcall_lambda (fun=XIL(0xa00000000612d0c8), nargs=1,
      arg_vector=0x82e370) at eval.c:2989
  #16 0x01384953 in Ffuncall (nargs=2, args=0x82e368) at eval.c:2796
  #17 0x01427e37 in exec_byte_code (bytestr=XIL(0x8000000006131a80),
      vector=XIL(0xa00000000612e450), maxdepth=make_fixnum(14),
      args_template=make_fixnum(0), nargs=0, args=0x82efa8) at bytecode.c:633
  #18 0x01385a11 in funcall_lambda (fun=XIL(0xa00000000612e420), nargs=0,
      arg_vector=0x82efa8) at eval.c:2989
  #19 0x01384953 in Ffuncall (nargs=1, args=0x82efa0) at eval.c:2796
  #20 0x01427e37 in exec_byte_code (bytestr=XIL(0x8000000006132488),
      vector=XIL(0xa000000006131c20), maxdepth=make_fixnum(12),
      args_template=make_fixnum(0), nargs=0, args=0x82f680) at bytecode.c:633
  #21 0x01385a11 in funcall_lambda (fun=XIL(0xa000000006131bf0), nargs=0,
      arg_vector=0x82f680) at eval.c:2989
  #22 0x01385519 in apply_lambda (fun=XIL(0xa000000006131bf0), args=XIL(0),
      count=4) at eval.c:2926
  #23 0x01382b51 in eval_sub (form=XIL(0xc000000006278198)) at eval.c:2318
  #24 0x013819e8 in Feval (form=XIL(0xc000000006278198), lexical=XIL(0))
      at eval.c:2102
  #25 0x011df7a3 in top_level_2 () at keyboard.c:1100
  #26 0x0137f02d in internal_condition_case (bfun=0x11df76d <top_level_2>,
      handlers=XIL(0x90), hfun=0x11def2f <cmd_error>) at eval.c:1355
  #27 0x011df818 in top_level_1 (ignore=XIL(0)) at keyboard.c:1108
  #28 0x0137e1f8 in internal_catch (tag=XIL(0xdfe0),
      func=0x11df7a9 <top_level_1>, arg=XIL(0)) at eval.c:1116
  #29 0x011df679 in command_loop () at keyboard.c:1069
  #30 0x011de9b7 in recursive_edit_1 () at keyboard.c:714
  #31 0x011dec2d in Frecursive_edit () at keyboard.c:786
  #32 0x011d3547 in main (argc=4, argv=0xa42848) at emacs.c:2054

  Lisp Backtrace:
  "dump-emacs-portable" (0x82d468)
  "progn" (0x82d658)
  "eval" (0x82d978)
  "command-line-1" (0x82e370)
  "command-line" (0x82efa8)
  "normal-top-level" (0x82f680)
  (gdb) fr 5
  #5  0x01327f92 in dump_object (ctx=0x82d1f0, object=XIL(0xa000000006288090))
      at pdumper.c:3127
  3127          offset = dump_vectorlike (ctx, object, offset);
  (gdb) p object
  $1 = XIL(0xa000000006288090)
  (gdb) xtype
  Lisp_Vectorlike
  PVEC_HASH_TABLE
  (gdb) xhashtable
  $2 = (struct Lisp_Hash_Table *) 0x6288090
  (gdb) p *$
  $3 = {
    header = {
      size = 1291882500
    },
    weak = XIL(0),
    hash = XIL(0xa0000000068192c0),
    next = XIL(0xa0000000068194d0),
    index = XIL(0xa0000000068196e0),
    count = 13,
    next_free = 13,
    purecopy = false,
    mutable = true,
    rehash_threshold = 0.8125,
    rehash_size = 0.5,
    key_and_value = XIL(0xa000000005fbe6f8),
    test = {
      name = XIL(0x5610),
      user_hash_function = XIL(0),
      user_cmp_function = XIL(0),
      cmpfn = 0x13a7ffe <cmpfn_equal>,
      hashfn = 0x13a8107 <hashfn_equal>
    },
    next_weak = 0x0
  }
  (gdb) p $2->key_and_value
  $4 = XIL(0xa000000005fbe6f8)
  (gdb) xtype
  Lisp_Vectorlike
  PVEC_NORMAL_VECTOR
  (gdb) xvector
  $5 = (struct Lisp_Vector *) 0x5fbe6f8
  0
  (gdb) p $2->test.name
  $6 = XIL(0x5610)
  (gdb) xtype
  Lisp_Symbol
  (gdb) xsymbol
  $7 = (struct Lisp_Symbol *) 0x1bc74d0 <lispsym+22032>
  "equal"
  (gdb) p expected_value
  $8 = XIL(0xa000000006a527d8)
  (gdb) xtype
  Lisp_Vectorlike
  PVEC_COMPILED
  (gdb) xcompiled
  $9 = (struct Lisp_Vector *) 0x6a527d8
  {make_fixnum(771), XIL(0x8000000006ab8250), XIL(0xa000000006a36558),
    make_fixnum(13), XIL(0x8000000006ab81c0)}
  (gdb) p found_value
  $10 = XIL(0xa000000006c2fc00)
  (gdb) xtype
  Lisp_Vectorlike
  PVEC_COMPILED
  (gdb) xcompiled
  $11 = (struct Lisp_Vector *) 0x6c2fc00
  {make_fixnum(771), XIL(0x80000000069d21b0), XIL(0xa000000006c2fba0),
    make_fixnum(13), XIL(0x8000000006a412b0)}
  (gdb)

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Pip Cet]

On Sun, Jan 5, 2020 at 6:46 PM Eli Zaretskii <eliz@gnu.org> wrote:
> In my debug build of Emacs 27.0.60 I get an assertion violation while
> dumping, which probably is already a sign of trouble.

I think that's unrelated, but here's some analysis anyway, in case we
want to fix this bug:

Different bytecode objects may be `equal', but have different sxhashes.

(equal #[0 "" [] 0]
       #[0 "" [] 0])
(sxhash #[0 "" [] 0])
(sxhash #[0 "" [] 0])

When such bytecodes are used as keys for a hash table using
hashfn_equal, I believe the result is, fairly obviously, two hash
table entries for equal keys; when the hash table is rehashed, we may
be unlucky enough to retrieve the wrong one, leading to the crash.

That's what cl-generic.el does for the table cl--generic-dispatchers,
which I believe is what you were looking at.

>   #2  0x01326817 in check_hash_table_rehash (table_orig=XIL(0xa000000006288090))
>       at pdumper.c:2684

Can you confirm table_orig is cl--generic-dispatchers?

Again, I doubt this is related to the original bug. That `equal'
behaves weirdly like this is a problem I've mentioned before (when
(equal a b) signals but (equal b a) succeeds), but the consensus then
was not to change it, so it's possible this is only the rehashing
check needing to be more careful.

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Eli Zaretskii]

> From: Pip Cet <pipcet@gmail.com>
> Date: Mon, 6 Jan 2020 15:51:57 +0000
> Cc: NiwTinray <niwtrx@icloud.com>, Daniel Colascione <dancol@dancol.org>, 38912@debbugs.gnu.org
> 
> >   #2  0x01326817 in check_hash_table_rehash (table_orig=XIL(0xa000000006288090))
> >       at pdumper.c:2684
> 
> Can you confirm table_orig is cl--generic-dispatchers?

I don't think I know how.  Hash tables don't seem to have names stored
in them, do they?  (If they did, I'd expect xhashtable to print that
name.)

> Again, I doubt this is related to the original bug. That `equal'
> behaves weirdly like this is a problem I've mentioned before (when
> (equal a b) signals but (equal b a) succeeds), but the consensus then
> was not to change it, so it's possible this is only the rehashing
> check needing to be more careful.

If you can suggest how to make that check more careful, please do.

Thanks.

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Pip Cet]

On Mon, Jan 6, 2020 at 4:33 PM Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Pip Cet <pipcet@gmail.com>
> > Date: Mon, 6 Jan 2020 15:51:57 +0000
> > Cc: NiwTinray <niwtrx@icloud.com>, Daniel Colascione <dancol@dancol.org>, 38912@debbugs.gnu.org
> >
> > >   #2  0x01326817 in check_hash_table_rehash (table_orig=XIL(0xa000000006288090))
> > >       at pdumper.c:2684
> >
> > Can you confirm table_orig is cl--generic-dispatchers?
>
> I don't think I know how.

I meant something like

p Fsymbol_value(intern("cl--generic-dispatchers"))

in gdb. Will that work on your platform?

> Hash tables don't seem to have names stored
> in them, do they?  (If they did, I'd expect xhashtable to print that
> name.)

They don't have names, no.

> > Again, I doubt this is related to the original bug. That `equal'
> > behaves weirdly like this is a problem I've mentioned before (when
> > (equal a b) signals but (equal b a) succeeds), but the consensus then
> > was not to change it, so it's possible this is only the rehashing
> > check needing to be more careful.
>
> If you can suggest how to make that check more careful, please do.

I'd suggest just skipping the entire check for hashfn_equal, at least
on the Emacs 27 branch. On master, I'd prefer to make equal-equality
imply sxhash-equality.

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Daniel Colascione]

>> From: Pip Cet <pipcet@gmail.com>
>> Date: Mon, 6 Jan 2020 15:51:57 +0000
>> Cc: NiwTinray <niwtrx@icloud.com>, Daniel Colascione
>> <dancol@dancol.org>, 38912@debbugs.gnu.org
>>
>> >   #2  0x01326817 in check_hash_table_rehash
>> (table_orig=XIL(0xa000000006288090))
>> >       at pdumper.c:2684
>>
>> Can you confirm table_orig is cl--generic-dispatchers?
>
> I don't think I know how.  Hash tables don't seem to have names stored
> in them, do they?  (If they did, I'd expect xhashtable to print that
> name.)
>
>> Again, I doubt this is related to the original bug. That `equal'
>> behaves weirdly like this is a problem I've mentioned before (when
>> (equal a b) signals but (equal b a) succeeds), but the consensus then
>> was not to change it, so it's possible this is only the rehashing
>> check needing to be more careful.

This consensus is wrong. We need to make equality imply sxhash equality or
very odd things (like this) will happen. C++ and Java both require this
relationship between hash code equality and object equality, and we should
too. Can we just fix the bytecode sxhash bug?

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Daniel Colascione]

> [Please use "Reply All" to reply, so that the bug address is kept on
> the CC list.]
>
>> From: NiwTinray <niwtrx@icloud.com>
>> Date: Sun, 5 Jan 2020 13:25:07 +0800
>>
>> > I cannot reproduce this from "emacs -Q" because 'use-package' is not a
>> > known function.  Can you show a recipe starting from "emacs -Q",
>> > please?
>>
>> Here. I've attached a minimal script file that helps reproduce this bug.
>>
>> (require 'package)
>> (package-initialize)
>> (add-to-list 'package-archives
>>              '("melpa-stable" . "https://stable.melpa.org/packages/") t)
>> (unless (package-installed-p 'evil)
>>   (package-refresh-contents)
>>   (package-install 'evil))
>> (require 'evil)
>> (dump-emacs-portable "/tmp/test.pdmp")
>>
>> The script downloads the package "evil" from Melpa stable, load the evil
>> package
>> and dumps an image to /tmp/test.pdmp.
>>
>> > Also, does this happen if you add -Q to the Emacs invocation after
>> > dumping?  If not, there's more detail missing in your report: the
>> > customizations in your init files.
>>
>>
>> Sure. Please download this file, and run the command:
>>
>> emacs --batch -Q --script evil.el
>>
>> To see the bug happen, load the test.pdmp file:
>>
>> emacs -Q --dump-file /tmp/test.pdmp
>>
>> You should see a segmentation fault:
>>
>> [1]    23369 segmentation fault (core dumped)  emacs -Q --dump-file
>> /tmp/test.pdmp
>>
>> I run debugger inside src/.gdbinit using the command:
>>
>> gdb -x .gdbinit --args ./emacs --dump-file /tmp/test.pdmp
>>
>> And logged backtrace. See my second attachment:
>>
>> (base) omnisky :: ~/emacs/src ‹emacs-27*› » gdb -x .gdbinit --args
>> ./emacs --dump-file /tmp/test.pdmp
>> GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
>> Copyright (C) 2016 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later
>> <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show
>> copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-linux-gnu".
>> Type "show configuration" for configuration details.
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>.
>> Find the GDB manual and other documentation resources online at:
>> <http://www.gnu.org/software/gdb/documentation/>.
>> For help, type "help".
>> Type "apropos word" to search for commands related to "word"...
>> Reading symbols from ./emacs...done.
>> warning: File "/home/ntr/emacs/src/.gdbinit" auto-loading has been
>> declined by your `auto-load safe-path' set to
>> "$debugdir:$datadir/auto-load".
>> To enable execution of this file add
>> 	add-auto-load-safe-path /home/ntr/emacs/src/.gdbinit
>> line to your configuration file "/home/ntr/.gdbinit".
>> To completely disable this security protection add
>> 	set auto-load safe-path /
>> line to your configuration file "/home/ntr/.gdbinit".
>> For more information about this security protection see the
>> "Auto-loading safe path" section in the GDB manual.  E.g., run from the
>> shell:
>> 	info "(gdb)Auto-loading safe path"
>> SIGINT is used by the debugger.
>> Are you sure you want to change it? (y or n) [answered Y; input not from
>> terminal]
>> Environment variable "DISPLAY" not defined.
>> TERM = xterm-24bits
>> Breakpoint 1 at 0x411df0: file emacs.c, line 370.
>> Breakpoint 2 at 0x4bfe60: file xterm.c, line 10130.
>> (gdb) r
>> Starting program: /home/ntr/emacs/src/emacs --dump-file /tmp/test.pdmp
>> /home/ntr/emacs/src/emacs:
>> /raid_sdc/home/ntr/anaconda3/lib/libtiff.so.5: no version information
>> available (required by /home/ntr/emacs/src/emacs)
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library
>> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x00000000004f12d0 in Fcurrent_active_maps (olp=olp@entry=XIL(0x30),
>> position=position@entry=XIL(0)) at keymap.c:1541
>> 1541	      && NILP (KVAR (current_kboard,
>> Voverriding_terminal_local_map))
>> (gdb) xbacktrace
>> "key-binding" (0xffffd5c8)
>> "turn-on-undo-tree-mode" (0xffffd758)
>> "global-undo-tree-mode-enable-in-buffers" (0xffffd948)
>> "run-hooks" (0xffffd9e8)
>> "run-mode-hooks" (0xffffdbc0)
>> "minibuffer-inactive-mode" (0xffffdd40)
>> (gdb)
>
> In my debug build of Emacs 27.0.60 I get an assertion violation while
> dumping, which probably is already a sign of trouble.
>
> Daniel, any ideas?

I haven't had a chance over the past few days to repro this problem, but I
hope to do so this weeekend. The messages about the assertion failure
*during* dumping do seem likely unrelated. The easiest way to debug this
particular crash is with rr. Run the original test as "rr record emacs
[args]", then run "rr replay". The latter will dump you in a GDB prompt.
Type "cont" and run the replay of Emacs until you get to the SIGSEGV. Run
"watch -l [expression]" to break out of execution whenever that value
changes, then run "reverse-cont" to run the replay *backwards* until you
get to the code that changed the variable with the bad value.

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Pip Cet]

Summary, since I'm CCing emacs-devel:

For bytecode objects, equal-equality does not imply sxhash-equality.
That led to an assertion failure in the pdumper code, even though the
pdumper code itself is apparently correct; it simply does not allow
for paradoxical behavior of hash tables with bytecode keys. (The
muffled sound you hear in the background is my trying not to point out
that eq-vs-eql leads to similar paradoxes).

On Mon, Jan 6, 2020 at 5:01 PM Daniel Colascione <dancol@dancol.org> wrote:
> >> Again, I doubt this is related to the original bug. That `equal'
> >> behaves weirdly like this is a problem I've mentioned before (when
> >> (equal a b) signals but (equal b a) succeeds), but the consensus then
> >> was not to change it, so it's possible this is only the rehashing
> >> check needing to be more careful.
> This consensus is wrong. We need to make equality imply sxhash equality or
> very odd things (like this) will happen.

I agree absolutely, of course, and maybe there's a consensus for
fixing equal for bytecodes but leaving it unfixed for cases in which
(equal a b) signals but (equal b a) does not.

> C++ and Java both require this
> relationship between hash code equality and object equality, and we should
> too.

We do document it, we just don't obey our documented API.

> Can we just fix the bytecode sxhash bug?

Is it too late to fix this for Emacs 27? If the answer is no, would a
good fix be to make equal=eq for bytecode objects? That should come
close to the old behavior, and we can fix things properly (by fixing
sxhash to hash byte code objects) on master.

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Eli Zaretskii]

> From: Pip Cet <pipcet@gmail.com>
> Date: Mon, 6 Jan 2020 16:38:48 +0000
> Cc: niwtrx@icloud.com, Daniel Colascione <dancol@dancol.org>, 38912@debbugs.gnu.org
> 
> > > Can you confirm table_orig is cl--generic-dispatchers?
> >
> > I don't think I know how.
> 
> I meant something like
> 
> p Fsymbol_value(intern("cl--generic-dispatchers"))
> 
> in gdb. Will that work on your platform?

Yes, of course:

  (gdb) p/x Fsymbol_value(intern("cl--generic-dispatchers"))
  $2 = 0xa000000006288090
  (gdb) p table_orig
  $3 = XIL(0xa000000006288090)

So yes, that's cl--generic-dispatchers.

> > If you can suggest how to make that check more careful, please do.
> 
> I'd suggest just skipping the entire check for hashfn_equal, at least
> on the Emacs 27 branch. On master, I'd prefer to make equal-equality
> imply sxhash-equality.

Patches will be welcome, thanks.

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Eli Zaretskii]

> Date: Mon, 6 Jan 2020 09:01:24 -0800
> From: "Daniel Colascione" <dancol@dancol.org>
> Cc: "Pip Cet" <pipcet@gmail.com>,
>  niwtrx@icloud.com,
>  dancol@dancol.org,
>  38912@debbugs.gnu.org
> 
> >> Again, I doubt this is related to the original bug. That `equal'
> >> behaves weirdly like this is a problem I've mentioned before (when
> >> (equal a b) signals but (equal b a) succeeds), but the consensus then
> >> was not to change it, so it's possible this is only the rehashing
> >> check needing to be more careful.
> 
> This consensus is wrong. We need to make equality imply sxhash equality or
> very odd things (like this) will happen. C++ and Java both require this
> relationship between hash code equality and object equality, and we should
> too. Can we just fix the bytecode sxhash bug?

That'd be fine with me, especially since the ELisp manual seems to say
this is indeed a bug.

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Eli Zaretskii]

> From: Pip Cet <pipcet@gmail.com>
> Date: Mon, 6 Jan 2020 17:13:40 +0000
> Cc: Eli Zaretskii <eliz@gnu.org>, niwtrx@icloud.com, 38912@debbugs.gnu.org
> 
> Summary, since I'm CCing emacs-devel:

Please don't, I added Stefan instead.

> > Can we just fix the bytecode sxhash bug?
> 
> Is it too late to fix this for Emacs 27?

Depends on the fix and its potential side effects.

OTOH, being able to pdump a customized Emacs is not a goal for Emacs
27.1 (we already know that it doesn't work 100%, and needs more work),
so if the fix is somewhat risky, we could do that on master.

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Stefan Monnier]

The problem is simply that `sxhash` doesn't use the same "rules" about
which objects are compared by identity and which objects are compared
by contents.

In `src/fns.c`, when we compare `internal_equal` and `sxhash`, we see
that `sxhash` only looks at the contents of vectorlikes if they are:

      BIGNUMP, VECTORP, RECORDP, or BOOL_VECTOR_P

whereas `internal_equal` looks inside many more vectorlikes:

	if (BIGNUMP (o1))
	  return mpz_cmp (*xbignum_val (o1), *xbignum_val (o2)) == 0;
	if (OVERLAYP (o1))
	  {
	    if (!internal_equal (OVERLAY_START (o1), OVERLAY_START (o2),
				 equal_kind, depth + 1, ht)
		|| !internal_equal (OVERLAY_END (o1), OVERLAY_END (o2),
				    equal_kind, depth + 1, ht))
	      return false;
	    o1 = XOVERLAY (o1)->plist;
	    o2 = XOVERLAY (o2)->plist;
	    depth++;
	    goto tail_recurse;
	  }
	if (MARKERP (o1))
	  {
	    return (XMARKER (o1)->buffer == XMARKER (o2)->buffer
		    && (XMARKER (o1)->buffer == 0
			|| XMARKER (o1)->bytepos == XMARKER (o2)->bytepos));
	  }
	/* Boolvectors are compared much like strings.  */
	if (BOOL_VECTOR_P (o1))
	  {
	    EMACS_INT size = bool_vector_size (o1);
	    if (size != bool_vector_size (o2))
	      return false;
	    if (memcmp (bool_vector_data (o1), bool_vector_data (o2),
			bool_vector_bytes (size)))
	      return false;
	    return true;
	  }
	if (WINDOW_CONFIGURATIONP (o1))
	  {
	    eassert (equal_kind != EQUAL_NO_QUIT);
	    return compare_window_configurations (o1, o2, false);
	  }

	/* Aside from them, only true vectors, char-tables, compiled
	   functions, and fonts (font-spec, font-entity, font-object)
	   are sensible to compare, so eliminate the others now.  */
	if (size & PSEUDOVECTOR_FLAG)
	  {
	    if (((size & PVEC_TYPE_MASK) >> PSEUDOVECTOR_AREA_BITS)
		< PVEC_COMPILED)
	      return false;
	    size &= PSEUDOVECTOR_SIZE_MASK;
	  }
	for (ptrdiff_t i = 0; i < size; i++)
	  {
	    Lisp_Object v1, v2;
	    v1 = AREF (o1, i);
	    v2 = AREF (o2, i);
	    if (!internal_equal (v1, v2, equal_kind, depth + 1, ht))
	      return false;
	  }
	return true;
      }
      break;

so the problem doesn't affect only byte-compiled objects but also
overlays, markers, windowconfigs, chartables, and fonts, AFAICT.

The fix should be to make `sxhash` follow the same rules as `internal_equal`.

This is a fairly long-standing problem, so unless it is newly triggered
in "normal" circumstances in Emacs-27, the fix is probably best on
`master`.


        Stefan

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Noam Postavsky]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> The problem is simply that `sxhash` doesn't use the same "rules" about
> which objects are compared by identity and which objects are compared
> by contents.

> The fix should be to make `sxhash` follow the same rules as `internal_equal`.
>
> This is a fairly long-standing problem, so unless it is newly triggered
> in "normal" circumstances in Emacs-27, the fix is probably best on
> `master`.

This was earlier reported as Bug#32503, by the way.

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Pip Cet <pipcet@gmail.com>,  niwtrx@icloud.com,  38912@debbugs.gnu.org,
>   dancol@dancol.org
> Date: Mon, 06 Jan 2020 13:13:31 -0500
> 
> This is a fairly long-standing problem, so unless it is newly triggered
> in "normal" circumstances in Emacs-27, the fix is probably best on
> `master`.

I tend to agree, since the capability of pdumping a customized Emacs
is from my POV not a goal for Emacs 27.1: we know that needs some more
work, and I wouldn't delay Emacs 27.1 to get it 100% right.

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Paul Eggert]

On 1/6/20 9:13 AM, Pip Cet wrote:
> we can fix things properly (by fixing
> sxhash to hash byte code objects) on master.

It's not just bytecode objects; it's also markers and some other stuff. 
The worst case is window configurations, where there's a nontrivial 
function compare_window_configurations that Fequal delegates to.

Does anyone know offhand why we don't simply use eq to compare window 
configurations? Might save me some work in patching this in master.

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[dancol]

[-- Attachment #1: Type: text/html, Size: 1040 bytes --]

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Stefan Monnier]

>  The worst case is window configurations, where there's a nontrivial 
>  function compare_window_configurations that Fequal delegates to.

Indeed, this is very strange.

>  Does anyone know offhand why we don't simply use eq to compare window 
>  configurations?

I'm curious about it too.  As a long-time Elisp coder, I find this
behavior very odd (I just discovered it while looking at the code)
and I think having them compared by identity would make a lot more sense.

> I think changing the equal behavior of these objects would be
> a compatibility change.

Indeed.  It's clearly out of the question for Emacs-27.  But I'd be in
favor of introducing this backward incompatibility in Emacs-28, unless
we can find a good use case where the structural-equality is needed
for them.


        Stefan

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Eli Zaretskii]

> Cc: Daniel Colascione <dancol@dancol.org>, emacs-devel@gnu.org,
>  niwtrx@icloud.com, 38912@debbugs.gnu.org, Eli Zaretskii <eliz@gnu.org>
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Mon, 6 Jan 2020 18:38:17 -0800
> 
> Does anyone know offhand why we don't simply use eq to compare window 
> configurations?

I guess because not all of that is Lisp data.  struct save_window_data
isn't.

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Stefan Monnier]

>> Does anyone know offhand why we don't simply use eq to compare window 
>> configurations?
> I guess because not all of that is Lisp data.  struct save_window_data
> isn't.

I think he meant: why doesn't `equal` applied on window configs behave
like `eq`?


        Stefan

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Paul Eggert <eggert@cs.ucla.edu>,  niwtrx@icloud.com,
>   38912@debbugs.gnu.org,  dancol@dancol.org,  pipcet@gmail.com,
>   emacs-devel@gnu.org
> Date: Tue, 07 Jan 2020 12:37:00 -0500
> 
> I think he meant: why doesn't `equal` applied on window configs behave
> like `eq`?

That's an entirely different question.

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Stefan Monnier]

>> I think he meant: why doesn't `equal` applied on window configs behave
>> like `eq`?
> That's an entirely different question.

Indeed, but I think that's the only question that's actually been asked
(Dan will hopefully correct me if I'm wrong).

I actually *expected* that `equal` on window configs would behave like
`eq`.  It would have never occurred to me (before I looked at the code)
that it could be different (although in retrospect, I can see reasons
why it could make sense).


        Stefan

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 07 Jan 2020 13:01:17 -0500
> Cc: niwtrx@icloud.com, eggert@cs.ucla.edu, emacs-devel@gnu.org,
>  38912@debbugs.gnu.org, pipcet@gmail.com, dancol@dancol.org
> 
> I actually *expected* that `equal` on window configs would behave like
> `eq`.  It would have never occurred to me (before I looked at the code)
> that it could be different (although in retrospect, I can see reasons
> why it could make sense).

Since we have a dedicated comparison function there, why is it
important what eq and equal do with these objects?

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[martin rudalics]

 > I actually *expected* that `equal` on window configs would behave like
 > `eq`.  It would have never occurred to me (before I looked at the code)
 > that it could be different (although in retrospect, I can see reasons
 > why it could make sense).

`eq' wouldn't make any sense at all.  'compare-window-configurations'
per se is a misconception.  But I have no intention to dig into
strokes.el to find out why on earth it would need such a thing and how
to remove it.

martin

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Stefan Monnier]

>> I actually *expected* that `equal` on window configs would behave like
>> `eq`.  It would have never occurred to me (before I looked at the code)
>> that it could be different (although in retrospect, I can see reasons
>> why it could make sense).
> `eq' wouldn't make any sense at all.

Why not?

> 'compare-window-configurations' per se is a misconception.

We're talking about the behavior of `equal`, not that of
`compare-window-configurations`.


        Stefan

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Stefan Monnier]

>> I actually *expected* that `equal` on window configs would behave like
>> `eq`.  It would have never occurred to me (before I looked at the code)
>> that it could be different (although in retrospect, I can see reasons
>> why it could make sense).
> Since we have a dedicated comparison function there, why is it
> important what eq and equal do with these objects?

The immediate reason why it's relevant is that it affects what `sxhash`
needs to do.  The existence of `compare-window-configurations` is not
directly relevant to the question (tho it arguably makes it less
important for `equal` to behave structurally since it provides an
alternative for those callers who would need it).


        Stefan

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[martin rudalics]

 >> `eq' wouldn't make any sense at all.
 >
 > Why not?

Where and how would you use it?

 >> 'compare-window-configurations' per se is a misconception.
 >
 > We're talking about the behavior of `equal`, not that of
 > `compare-window-configurations`.

'equal' is specified via compare_window_configurations which is a
misconception just a 'compare-window-configurations' is.

martin

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Pip Cet]

On Mon, Jan 6, 2020 at 6:13 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> The problem is simply that `sxhash` doesn't use the same "rules" about
> which objects are compared by identity and which objects are compared
> by contents.

I agree.

> so the problem doesn't affect only byte-compiled objects but also
> overlays, markers, windowconfigs, chartables, and fonts, AFAICT.

True, but then, those don't make much sense as hash keys in an
equal-based hash table. I don't think bytecode objects make sense as
hash keys in an equal-based hash table, either: two bytecode objects
can be `equal` yet be non-equivalent, even though it's somewhat
unlikely to occur in practice.

> The fix should be to make `sxhash` follow the same rules as `internal_equal`.

I think the two should largely follow the same rules, because all of
those objects should be equal iff they're eq. Hashing a marker based
on its position, for example, makes little sense: when the marker
moves, the hash value would change. Bytecode objects could be compared
based on their string representation, I guess, and maybe should be
hash-consed based on that (deduplicating identical bytecode objects
used as constants in other bytecode objects appears to save quite a
bit of space), once they're made immutable.

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1302 bytes --]

On 1/7/20 6:16 AM, Stefan Monnier wrote:
> I'd be in
> favor of introducing this backward incompatibility in Emacs-28, unless
> we can find a good use case where the structural-equality is needed
> for them.

I looked for use cases within Emacs, and found only one: a test case 
that I changed to use compare-window-configurations instead. I don't 
think this was a good use case; it's just a hack in a test.

I agree with Martin that comparing window configurations is something of 
a botch. Despite some documentation to the contrary, neither 'equal' nor 
compare-window-configurations compares all components of window 
configurations (though the two functions do compare different sets of 
components). If there's a solid need for comparing different sets of 
components of window configurations (and I doubt whether the test case 
establishes a need), it could be implemented by adding an argument to 
compare-window-configurations to say how to do the comparison. However, 
there doesn't seem to be a significant need to make all this part of 
'equal'.

So, after writing some sxhash-equal test cases and then fixing Emacs to 
handle these test cases properly, I boldly installed the attached patch 
into master. This should make sxhash-equal compatible with 'equal'. 
Comments welcome as usual.

[-- Attachment #2: 0001-Fix-sxhash-equal-on-bytecodes-markers-etc.patch --]
[-- Type: text/x-patch, Size: 12564 bytes --]

From 724af7671590cd91df37f64df6be73f6dca0144d Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Tue, 7 Jan 2020 11:23:11 -0800
Subject: [PATCH] Fix sxhash-equal on bytecodes, markers, etc.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Problem reported by Pip Cet (Bug#38912#14).
* doc/lispref/objects.texi (Equality Predicates):
Document better when ‘equal’ looks inside objects.
* doc/lispref/windows.texi (Window Configurations):
Don’t say that ‘equal’ looks inside window configurations.
* etc/NEWS: Mention the change.
* src/fns.c (internal_equal):
Do not look inside window configurations.
(sxhash_obj): Hash markers, byte-code function objects,
char-tables, and font objects consistently with Fequal.
* src/window.c (compare_window_configurations):
Now static.  Remove last argument.  Caller changed.
* test/lisp/ffap-tests.el (ffap-other-window--bug-25352):
Use compare-window-configurations, not ‘equal’.
* test/src/fns-tests.el (test-sxhash-equal): New test.
---
 doc/lispref/objects.texi |  8 +++++--
 doc/lispref/windows.texi |  4 ----
 etc/NEWS                 |  6 +++++
 src/fns.c                | 52 ++++++++++++++++++++++++----------------
 src/lisp.h               |  4 ++--
 src/window.c             | 21 ++++------------
 src/window.h             |  1 -
 test/lisp/ffap-tests.el  |  2 +-
 test/src/fns-tests.el    | 16 +++++++++++++
 9 files changed, 67 insertions(+), 47 deletions(-)

diff --git a/doc/lispref/objects.texi b/doc/lispref/objects.texi
index 4be2eb6918..4242223a48 100644
--- a/doc/lispref/objects.texi
+++ b/doc/lispref/objects.texi
@@ -2336,8 +2336,12 @@ Equality Predicates
 @end group
 @end example
 
-However, two distinct buffers are never considered @code{equal}, even if
-their textual contents are the same.
+The @code{equal} function recursively compares the contents of objects
+if they are integers, strings, markers, vectors, bool-vectors,
+byte-code function objects, char-tables, records, or font objects.
+Other objects are considered @code{equal} only if they are @code{eq}.
+For example, two distinct buffers are never considered @code{equal},
+even if their textual contents are the same.
 @end defun
 
   For @code{equal}, equality is defined recursively; for example, given
diff --git a/doc/lispref/windows.texi b/doc/lispref/windows.texi
index c9301c9d18..d0791d4019 100644
--- a/doc/lispref/windows.texi
+++ b/doc/lispref/windows.texi
@@ -5915,10 +5915,6 @@ Window Configurations
 structure of windows, but ignores the values of point and the
 saved scrolling positions---it can return @code{t} even if those
 aspects differ.
-
-The function @code{equal} can also compare two window configurations; it
-regards configurations as unequal if they differ in any respect, even a
-saved point.
 @end defun
 
 @defun window-configuration-frame config
diff --git a/etc/NEWS b/etc/NEWS
index d6cabf8e9e..0784160ce2 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -42,6 +42,12 @@ applies, and please also update docstrings as needed.
 \f
 * Incompatible Lisp Changes in Emacs 28.1
 
+** 'equal' no longer examines some contents of window configurations.
+Instead, it considers window configurations to be equal only if they
+are eq.  To compare contents, use compare-window-configurations
+instead.  This change helps fix a bug in sxhash-equal, which returned
+incorrect hashes for window configurations and some other objects.
+
 \f
 * Lisp Changes in Emacs 28.1
 
diff --git a/src/fns.c b/src/fns.c
index 4a0a8fd96d..4a463a8feb 100644
--- a/src/fns.c
+++ b/src/fns.c
@@ -2434,6 +2434,9 @@ internal_equal (Lisp_Object o1, Lisp_Object o2, enum equal_kind equal_kind,
 	   same size.  */
 	if (ASIZE (o2) != size)
 	  return false;
+
+	/* Compare bignums, overlays, markers, and boolvectors
+	   specially, by comparing their values.  */
 	if (BIGNUMP (o1))
 	  return mpz_cmp (*xbignum_val (o1), *xbignum_val (o2)) == 0;
 	if (OVERLAYP (o1))
@@ -2454,7 +2457,6 @@ internal_equal (Lisp_Object o1, Lisp_Object o2, enum equal_kind equal_kind,
 		    && (XMARKER (o1)->buffer == 0
 			|| XMARKER (o1)->bytepos == XMARKER (o2)->bytepos));
 	  }
-	/* Boolvectors are compared much like strings.  */
 	if (BOOL_VECTOR_P (o1))
 	  {
 	    EMACS_INT size = bool_vector_size (o1);
@@ -2465,11 +2467,6 @@ internal_equal (Lisp_Object o1, Lisp_Object o2, enum equal_kind equal_kind,
 	      return false;
 	    return true;
 	  }
-	if (WINDOW_CONFIGURATIONP (o1))
-	  {
-	    eassert (equal_kind != EQUAL_NO_QUIT);
-	    return compare_window_configurations (o1, o2, false);
-	  }
 
 	/* Aside from them, only true vectors, char-tables, compiled
 	   functions, and fonts (font-spec, font-entity, font-object)
@@ -4703,22 +4700,35 @@ sxhash_obj (Lisp_Object obj, int depth)
       hash = sxhash_string (SSDATA (obj), SBYTES (obj));
       break;
 
-      /* This can be everything from a vector to an overlay.  */
     case Lisp_Vectorlike:
-      if (BIGNUMP (obj))
-	hash = sxhash_bignum (obj);
-      else if (VECTORP (obj) || RECORDP (obj))
-	/* According to the CL HyperSpec, two arrays are equal only if
-	   they are `eq', except for strings and bit-vectors.  In
-	   Emacs, this works differently.  We have to compare element
-	   by element.  Same for records.  */
-	hash = sxhash_vector (obj, depth);
-      else if (BOOL_VECTOR_P (obj))
-	hash = sxhash_bool_vector (obj);
-      else
-	/* Others are `equal' if they are `eq', so let's take their
-	   address as hash.  */
-	hash = XHASH (obj);
+      {
+	enum pvec_type pvec_type = PSEUDOVECTOR_TYPE (XVECTOR (obj));
+	if (! (PVEC_NORMAL_VECTOR < pvec_type && pvec_type < PVEC_COMPILED))
+	  {
+	    /* According to the CL HyperSpec, two arrays are equal only if
+	       they are 'eq', except for strings and bit-vectors.  In
+	       Emacs, this works differently.  We have to compare element
+	       by element.  Same for pseudovectors that internal_equal
+	       examines the Lisp contents of.  */
+	    hash = sxhash_vector (obj, depth);
+	    break;
+	  }
+	else if (pvec_type == PVEC_BIGNUM)
+	  hash = sxhash_bignum (obj);
+	else if (pvec_type == PVEC_MARKER)
+	  {
+	    ptrdiff_t bytepos
+	      = XMARKER (obj)->buffer ? XMARKER (obj)->bytepos : 0;
+	    hash = sxhash_combine ((intptr_t) XMARKER (obj)->buffer, bytepos);
+	    hash = SXHASH_REDUCE (hash);
+	  }
+	else if (pvec_type == PVEC_BOOL_VECTOR)
+	  hash = sxhash_bool_vector (obj);
+	else
+	  /* Others are 'equal' if they are 'eq', so take their
+	     address as hash.  */
+	  hash = XHASH (obj);
+      }
       break;
 
     case Lisp_Cons:
diff --git a/src/lisp.h b/src/lisp.h
index 1a1ae0399b..3681b7b2a7 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -1069,7 +1069,7 @@ DEFINE_GDB_SYMBOL_END (PSEUDOVECTOR_FLAG)
    with PVEC_TYPE_MASK to indicate the actual type.  */
 enum pvec_type
 {
-  PVEC_NORMAL_VECTOR,
+  PVEC_NORMAL_VECTOR,	/* Should be first, for sxhash_obj.  */
   PVEC_FREE,
   PVEC_BIGNUM,
   PVEC_MARKER,
@@ -1094,7 +1094,7 @@ DEFINE_GDB_SYMBOL_END (PSEUDOVECTOR_FLAG)
   PVEC_CONDVAR,
   PVEC_MODULE_FUNCTION,
 
-  /* These should be last, check internal_equal to see why.  */
+  /* These should be last, for internal_equal and sxhash_obj.  */
   PVEC_COMPILED,
   PVEC_CHAR_TABLE,
   PVEC_SUB_CHAR_TABLE,
diff --git a/src/window.c b/src/window.c
index ff17cd88f3..8cdad27b66 100644
--- a/src/window.c
+++ b/src/window.c
@@ -7976,19 +7976,17 @@ foreach_window_1 (struct window *w, bool (*fn) (struct window *, void *),
 /* Return true if window configurations CONFIGURATION1 and CONFIGURATION2
    describe the same state of affairs.  This is used by Fequal.
 
-   IGNORE_POSITIONS means ignore non-matching scroll positions
-   and the like.
+   Ignore non-matching scroll positions and the like.
 
    This ignores a couple of things like the dedication status of
    window, combination_limit and the like.  This might have to be
    fixed.  */
 
-bool
+static bool
 compare_window_configurations (Lisp_Object configuration1,
-			       Lisp_Object configuration2,
-			       bool ignore_positions)
+			       Lisp_Object configuration2)
 {
-  register struct save_window_data *d1, *d2;
+  struct save_window_data *d1, *d2;
   struct Lisp_Vector *sws1, *sws2;
   ptrdiff_t i;
 
@@ -8006,9 +8004,6 @@ compare_window_configurations (Lisp_Object configuration1,
       || d1->frame_menu_bar_lines != d2->frame_menu_bar_lines
       || !EQ (d1->selected_frame, d2->selected_frame)
       || !EQ (d1->f_current_buffer, d2->f_current_buffer)
-      || (!ignore_positions
-	  && (!EQ (d1->minibuf_scroll_window, d2->minibuf_scroll_window)
-	      || !EQ (d1->minibuf_selected_window, d2->minibuf_selected_window)))
       || !EQ (d1->focus_frame, d2->focus_frame)
       /* Verify that the two configurations have the same number of windows.  */
       || sws1->header.size != sws2->header.size)
@@ -8041,12 +8036,6 @@ compare_window_configurations (Lisp_Object configuration1,
 	     equality.  */
 	  || !EQ (sw1->parent, sw2->parent)
 	  || !EQ (sw1->prev, sw2->prev)
-	  || (!ignore_positions
-	      && (!EQ (sw1->hscroll, sw2->hscroll)
-		  || !EQ (sw1->min_hscroll, sw2->min_hscroll)
-		  || !EQ (sw1->start_at_line_beg, sw2->start_at_line_beg)
-		  || NILP (Fequal (sw1->start, sw2->start))
-		  || NILP (Fequal (sw1->pointm, sw2->pointm))))
 	  || !EQ (sw1->left_margin_cols, sw2->left_margin_cols)
 	  || !EQ (sw1->right_margin_cols, sw2->right_margin_cols)
 	  || !EQ (sw1->left_fringe_width, sw2->left_fringe_width)
@@ -8071,7 +8060,7 @@ DEFUN ("compare-window-configurations", Fcompare_window_configurations,
 and scrolling positions.  */)
   (Lisp_Object x, Lisp_Object y)
 {
-  if (compare_window_configurations (x, y, true))
+  if (compare_window_configurations (x, y))
     return Qt;
   return Qnil;
 }
diff --git a/src/window.h b/src/window.h
index aa8d2c8d1d..167d1be7ab 100644
--- a/src/window.h
+++ b/src/window.h
@@ -1184,7 +1184,6 @@ #define CHECK_LIVE_WINDOW(WINDOW)				\
 extern Lisp_Object window_parameter (struct window *, Lisp_Object parameter);
 extern struct window *decode_live_window (Lisp_Object);
 extern struct window *decode_any_window (Lisp_Object);
-extern bool compare_window_configurations (Lisp_Object, Lisp_Object, bool);
 extern void mark_window_cursors_off (struct window *);
 extern bool window_wants_mode_line (struct window *);
 extern bool window_wants_header_line (struct window *);
diff --git a/test/lisp/ffap-tests.el b/test/lisp/ffap-tests.el
index eaf39680e4..30c8f79457 100644
--- a/test/lisp/ffap-tests.el
+++ b/test/lisp/ffap-tests.el
@@ -74,7 +74,7 @@ ffap-other-window--bug-25352
              (urls nil)
              (ffap-url-fetcher (lambda (url) (push url urls) nil)))
     (should-not (ffap-other-window "https://www.gnu.org"))
-    (should (equal (current-window-configuration) old))
+    (should (compare-window-configurations (current-window-configuration) old))
     (should (equal urls '("https://www.gnu.org")))))
 
 (provide 'ffap-tests)
diff --git a/test/src/fns-tests.el b/test/src/fns-tests.el
index 60be2c6c2d..c6ceae4a00 100644
--- a/test/src/fns-tests.el
+++ b/test/src/fns-tests.el
@@ -858,6 +858,22 @@ test-hash-function-that-mutates-hash-table
        (puthash k k h)))
     (should (= 100 (hash-table-count h)))))
 
+(ert-deftest test-sxhash-equal ()
+  (should (= (sxhash-equal (* most-positive-fixnum most-negative-fixnum))
+	     (sxhash-equal (* most-positive-fixnum most-negative-fixnum))))
+  (should (= (sxhash-equal (make-string 1000 ?a))
+	     (sxhash-equal (make-string 1000 ?a))))
+  (should (= (sxhash-equal (point-marker))
+	     (sxhash-equal (point-marker))))
+  (should (= (sxhash-equal (make-vector 1000 (make-string 10 ?a)))
+	     (sxhash-equal (make-vector 1000 (make-string 10 ?a)))))
+  (should (= (sxhash-equal (make-bool-vector 1000 t))
+	     (sxhash-equal (make-bool-vector 1000 t))))
+  (should (= (sxhash-equal (make-char-table nil (make-string 10 ?a)))
+	     (sxhash-equal (make-char-table nil (make-string 10 ?a)))))
+  (should (= (sxhash-equal (record 'a (make-string 10 ?a)))
+	     (sxhash-equal (record 'a (make-string 10 ?a))))))
+
 (ert-deftest test-secure-hash ()
   (should (equal (secure-hash 'md5    "foobar")
                  "3858f62230ac3c915f300c664312c63f"))
-- 
2.24.1

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Stefan Monnier]

>> so the problem doesn't affect only byte-compiled objects but also
>> overlays, markers, windowconfigs, chartables, and fonts, AFAICT.
> True, but then, those don't make much sense as hash keys in an
> equal-based hash table.

Comparing markers with `equal` does make some kind of sense.
And `sxhash-equal` needs to be consistent with it, regardless if we
think it makes sense to index an equal-hash-table with markers (or
objects containing markers).

> I don't think bytecode objects make sense as
> hash keys in an equal-based hash table,

I agree that comparing function objects is fundamentally unreliable, yet
we do that in several places in Elisp: it's not super common, but it's
not rare either, and having a better approximation of "behaves
identically" than `eq` is occasionally useful.

> on its position, for example, makes little sense: when the marker
> moves, the hash value would change.

Comparison with `equal` suffers from this problem in many more cases
than just markers (cons cells, vectors, strings, ...).

> (deduplicating identical bytecode objects used as constants in other
> bytecode objects appears to save quite a bit of space),

I'm curious where you've seen that (I haven't looked at it, but indeed
it's quite possible that the byte-compiler "often" ends up generating
identical byte objects).


        Stefan

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This consensus is wrong. We need to make equality imply sxhash equality or
  > very odd things (like this) will happen.

I agree.  The basic idea of sxhash is that comparing hashes is a quick
way to detect most cases where two values are not equal.  If sxhash
comparison gives false negatives, that method is not trustworthy.

-- 
Dr Richard Stallman
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Eli Zaretskii]

Ping!  Any news in debugging this?

> Date: Mon, 6 Jan 2020 09:10:20 -0800
> From: "Daniel Colascione" <dancol@dancol.org>
> Cc: "NiwTinray" <niwtrx@icloud.com>,
>  "Daniel Colascione" <dancol@dancol.org>,
>  38912@debbugs.gnu.org
> 
> > [Please use "Reply All" to reply, so that the bug address is kept on
> > the CC list.]
> >
> >> From: NiwTinray <niwtrx@icloud.com>
> >> Date: Sun, 5 Jan 2020 13:25:07 +0800
> >>
> >> > I cannot reproduce this from "emacs -Q" because 'use-package' is not a
> >> > known function.  Can you show a recipe starting from "emacs -Q",
> >> > please?
> >>
> >> Here. I've attached a minimal script file that helps reproduce this bug.
> >>
> >> (require 'package)
> >> (package-initialize)
> >> (add-to-list 'package-archives
> >>              '("melpa-stable" . "https://stable.melpa.org/packages/") t)
> >> (unless (package-installed-p 'evil)
> >>   (package-refresh-contents)
> >>   (package-install 'evil))
> >> (require 'evil)
> >> (dump-emacs-portable "/tmp/test.pdmp")
> >>
> >> The script downloads the package "evil" from Melpa stable, load the evil
> >> package
> >> and dumps an image to /tmp/test.pdmp.
> >>
> >> > Also, does this happen if you add -Q to the Emacs invocation after
> >> > dumping?  If not, there's more detail missing in your report: the
> >> > customizations in your init files.
> >>
> >>
> >> Sure. Please download this file, and run the command:
> >>
> >> emacs --batch -Q --script evil.el
> >>
> >> To see the bug happen, load the test.pdmp file:
> >>
> >> emacs -Q --dump-file /tmp/test.pdmp
> >>
> >> You should see a segmentation fault:
> >>
> >> [1]    23369 segmentation fault (core dumped)  emacs -Q --dump-file
> >> /tmp/test.pdmp
> >>
> >> I run debugger inside src/.gdbinit using the command:
> >>
> >> gdb -x .gdbinit --args ./emacs --dump-file /tmp/test.pdmp
> >>
> >> And logged backtrace. See my second attachment:
> >>
> >> (base) omnisky :: ~/emacs/src ‹emacs-27*› » gdb -x .gdbinit --args
> >> ./emacs --dump-file /tmp/test.pdmp
> >> GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
> >> Copyright (C) 2016 Free Software Foundation, Inc.
> >> License GPLv3+: GNU GPL version 3 or later
> >> <http://gnu.org/licenses/gpl.html>
> >> This is free software: you are free to change and redistribute it.
> >> There is NO WARRANTY, to the extent permitted by law.  Type "show
> >> copying"
> >> and "show warranty" for details.
> >> This GDB was configured as "x86_64-linux-gnu".
> >> Type "show configuration" for configuration details.
> >> For bug reporting instructions, please see:
> >> <http://www.gnu.org/software/gdb/bugs/>.
> >> Find the GDB manual and other documentation resources online at:
> >> <http://www.gnu.org/software/gdb/documentation/>.
> >> For help, type "help".
> >> Type "apropos word" to search for commands related to "word"...
> >> Reading symbols from ./emacs...done.
> >> warning: File "/home/ntr/emacs/src/.gdbinit" auto-loading has been
> >> declined by your `auto-load safe-path' set to
> >> "$debugdir:$datadir/auto-load".
> >> To enable execution of this file add
> >> 	add-auto-load-safe-path /home/ntr/emacs/src/.gdbinit
> >> line to your configuration file "/home/ntr/.gdbinit".
> >> To completely disable this security protection add
> >> 	set auto-load safe-path /
> >> line to your configuration file "/home/ntr/.gdbinit".
> >> For more information about this security protection see the
> >> "Auto-loading safe path" section in the GDB manual.  E.g., run from the
> >> shell:
> >> 	info "(gdb)Auto-loading safe path"
> >> SIGINT is used by the debugger.
> >> Are you sure you want to change it? (y or n) [answered Y; input not from
> >> terminal]
> >> Environment variable "DISPLAY" not defined.
> >> TERM = xterm-24bits
> >> Breakpoint 1 at 0x411df0: file emacs.c, line 370.
> >> Breakpoint 2 at 0x4bfe60: file xterm.c, line 10130.
> >> (gdb) r
> >> Starting program: /home/ntr/emacs/src/emacs --dump-file /tmp/test.pdmp
> >> /home/ntr/emacs/src/emacs:
> >> /raid_sdc/home/ntr/anaconda3/lib/libtiff.so.5: no version information
> >> available (required by /home/ntr/emacs/src/emacs)
> >> [Thread debugging using libthread_db enabled]
> >> Using host libthread_db library
> >> "/lib/x86_64-linux-gnu/libthread_db.so.1".
> >>
> >> Program received signal SIGSEGV, Segmentation fault.
> >> 0x00000000004f12d0 in Fcurrent_active_maps (olp=olp@entry=XIL(0x30),
> >> position=position@entry=XIL(0)) at keymap.c:1541
> >> 1541	      && NILP (KVAR (current_kboard,
> >> Voverriding_terminal_local_map))
> >> (gdb) xbacktrace
> >> "key-binding" (0xffffd5c8)
> >> "turn-on-undo-tree-mode" (0xffffd758)
> >> "global-undo-tree-mode-enable-in-buffers" (0xffffd948)
> >> "run-hooks" (0xffffd9e8)
> >> "run-mode-hooks" (0xffffdbc0)
> >> "minibuffer-inactive-mode" (0xffffdd40)
> >> (gdb)
> >
> > In my debug build of Emacs 27.0.60 I get an assertion violation while
> > dumping, which probably is already a sign of trouble.
> >
> > Daniel, any ideas?
> 
> I haven't had a chance over the past few days to repro this problem, but I
> hope to do so this weeekend. The messages about the assertion failure
> *during* dumping do seem likely unrelated. The easiest way to debug this
> particular crash is with rr. Run the original test as "rr record emacs
> [args]", then run "rr replay". The latter will dump you in a GDB prompt.
> Type "cont" and run the replay of Emacs until you get to the SIGSEGV. Run
> "watch -l [expression]" to break out of execution whenever that value
> changes, then run "reverse-cont" to run the replay *backwards* until you
> get to the code that changed the variable with the bad value.
> 
> 

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Daniel Colascione]

Sorry, haven't had a chance to look at it yet. I've been treating it as
low-ish priority because pdumping outside loadup isn't supported yet. Is
there some reason to expedite this work?

> Ping!  Any news in debugging this?
>
>> Date: Mon, 6 Jan 2020 09:10:20 -0800
>> From: "Daniel Colascione" <dancol@dancol.org>
>> Cc: "NiwTinray" <niwtrx@icloud.com>,
>>  "Daniel Colascione" <dancol@dancol.org>,
>>  38912@debbugs.gnu.org
>>
>> > [Please use "Reply All" to reply, so that the bug address is kept on
>> > the CC list.]
>> >
>> >> From: NiwTinray <niwtrx@icloud.com>
>> >> Date: Sun, 5 Jan 2020 13:25:07 +0800
>> >>
>> >> > I cannot reproduce this from "emacs -Q" because 'use-package' is
>> not a
>> >> > known function.  Can you show a recipe starting from "emacs -Q",
>> >> > please?
>> >>
>> >> Here. I've attached a minimal script file that helps reproduce this
>> bug.
>> >>
>> >> (require 'package)
>> >> (package-initialize)
>> >> (add-to-list 'package-archives
>> >>              '("melpa-stable" . "https://stable.melpa.org/packages/")
>> t)
>> >> (unless (package-installed-p 'evil)
>> >>   (package-refresh-contents)
>> >>   (package-install 'evil))
>> >> (require 'evil)
>> >> (dump-emacs-portable "/tmp/test.pdmp")
>> >>
>> >> The script downloads the package "evil" from Melpa stable, load the
>> evil
>> >> package
>> >> and dumps an image to /tmp/test.pdmp.
>> >>
>> >> > Also, does this happen if you add -Q to the Emacs invocation after
>> >> > dumping?  If not, there's more detail missing in your report: the
>> >> > customizations in your init files.
>> >>
>> >>
>> >> Sure. Please download this file, and run the command:
>> >>
>> >> emacs --batch -Q --script evil.el
>> >>
>> >> To see the bug happen, load the test.pdmp file:
>> >>
>> >> emacs -Q --dump-file /tmp/test.pdmp
>> >>
>> >> You should see a segmentation fault:
>> >>
>> >> [1]    23369 segmentation fault (core dumped)  emacs -Q --dump-file
>> >> /tmp/test.pdmp
>> >>
>> >> I run debugger inside src/.gdbinit using the command:
>> >>
>> >> gdb -x .gdbinit --args ./emacs --dump-file /tmp/test.pdmp
>> >>
>> >> And logged backtrace. See my second attachment:
>> >>
>> >> (base) omnisky :: ~/emacs/src ‹emacs-27*› » gdb -x .gdbinit
>> --args
>> >> ./emacs --dump-file /tmp/test.pdmp
>> >> GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
>> >> Copyright (C) 2016 Free Software Foundation, Inc.
>> >> License GPLv3+: GNU GPL version 3 or later
>> >> <http://gnu.org/licenses/gpl.html>
>> >> This is free software: you are free to change and redistribute it.
>> >> There is NO WARRANTY, to the extent permitted by law.  Type "show
>> >> copying"
>> >> and "show warranty" for details.
>> >> This GDB was configured as "x86_64-linux-gnu".
>> >> Type "show configuration" for configuration details.
>> >> For bug reporting instructions, please see:
>> >> <http://www.gnu.org/software/gdb/bugs/>.
>> >> Find the GDB manual and other documentation resources online at:
>> >> <http://www.gnu.org/software/gdb/documentation/>.
>> >> For help, type "help".
>> >> Type "apropos word" to search for commands related to "word"...
>> >> Reading symbols from ./emacs...done.
>> >> warning: File "/home/ntr/emacs/src/.gdbinit" auto-loading has been
>> >> declined by your `auto-load safe-path' set to
>> >> "$debugdir:$datadir/auto-load".
>> >> To enable execution of this file add
>> >> 	add-auto-load-safe-path /home/ntr/emacs/src/.gdbinit
>> >> line to your configuration file "/home/ntr/.gdbinit".
>> >> To completely disable this security protection add
>> >> 	set auto-load safe-path /
>> >> line to your configuration file "/home/ntr/.gdbinit".
>> >> For more information about this security protection see the
>> >> "Auto-loading safe path" section in the GDB manual.  E.g., run from
>> the
>> >> shell:
>> >> 	info "(gdb)Auto-loading safe path"
>> >> SIGINT is used by the debugger.
>> >> Are you sure you want to change it? (y or n) [answered Y; input not
>> from
>> >> terminal]
>> >> Environment variable "DISPLAY" not defined.
>> >> TERM = xterm-24bits
>> >> Breakpoint 1 at 0x411df0: file emacs.c, line 370.
>> >> Breakpoint 2 at 0x4bfe60: file xterm.c, line 10130.
>> >> (gdb) r
>> >> Starting program: /home/ntr/emacs/src/emacs --dump-file
>> /tmp/test.pdmp
>> >> /home/ntr/emacs/src/emacs:
>> >> /raid_sdc/home/ntr/anaconda3/lib/libtiff.so.5: no version information
>> >> available (required by /home/ntr/emacs/src/emacs)
>> >> [Thread debugging using libthread_db enabled]
>> >> Using host libthread_db library
>> >> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> >>
>> >> Program received signal SIGSEGV, Segmentation fault.
>> >> 0x00000000004f12d0 in Fcurrent_active_maps (olp=olp@entry=XIL(0x30),
>> >> position=position@entry=XIL(0)) at keymap.c:1541
>> >> 1541	      && NILP (KVAR (current_kboard,
>> >> Voverriding_terminal_local_map))
>> >> (gdb) xbacktrace
>> >> "key-binding" (0xffffd5c8)
>> >> "turn-on-undo-tree-mode" (0xffffd758)
>> >> "global-undo-tree-mode-enable-in-buffers" (0xffffd948)
>> >> "run-hooks" (0xffffd9e8)
>> >> "run-mode-hooks" (0xffffdbc0)
>> >> "minibuffer-inactive-mode" (0xffffdd40)
>> >> (gdb)
>> >
>> > In my debug build of Emacs 27.0.60 I get an assertion violation while
>> > dumping, which probably is already a sign of trouble.
>> >
>> > Daniel, any ideas?
>>
>> I haven't had a chance over the past few days to repro this problem, but
>> I
>> hope to do so this weeekend. The messages about the assertion failure
>> *during* dumping do seem likely unrelated. The easiest way to debug this
>> particular crash is with rr. Run the original test as "rr record emacs
>> [args]", then run "rr replay". The latter will dump you in a GDB prompt.
>> Type "cont" and run the replay of Emacs until you get to the SIGSEGV.
>> Run
>> "watch -l [expression]" to break out of execution whenever that value
>> changes, then run "reverse-cont" to run the replay *backwards* until you
>> get to the code that changed the variable with the bad value.
>>
>>
>

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Eli Zaretskii]

> Date: Sun, 8 Mar 2020 19:15:31 -0700
> From: "Daniel Colascione" <dancol@dancol.org>
> Cc: "Daniel Colascione" <dancol@dancol.org>,
>  niwtrx@icloud.com,
>  38912@debbugs.gnu.org
> 
> Sorry, haven't had a chance to look at it yet. I've been treating it as
> low-ish priority because pdumping outside loadup isn't supported yet. Is
> there some reason to expedite this work?

Not that I can see, no.  I just wanted to be sure this isn't
forgotten.

Thanks.

bug#32503: 26.1; Byte-compiled functions don't hash consistently

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Sorry, haven't had a chance to look at it yet. I've been treating it as
>> low-ish priority because pdumping outside loadup isn't supported yet. Is
>> there some reason to expedite this work?
>
> Not that I can see, no.  I just wanted to be sure this isn't
> forgotten.
>
> Thanks.

This was a long thread, and I only skimmed it lightly.  But I noticed
that it was merged with this bug report:

Adam Porter <adam@alphapapa.net> writes:

> I noticed that byte-compiled functions don't hash consistently.  Here's
> an ECM from Noam Postavsky
> <https://github.com/skeeto/emacs-memoize/pull/10#issuecomment-415066682>:
>
>     (let ((obj1 (byte-compile (lambda (x) x)))
>           (obj2 (byte-compile (lambda (x) x))))
>       (list (equal obj1 obj2)
>             (eq obj1 obj2)
>             (= (sxhash obj1)
>                (sxhash obj2))))
>     ;=> (t nil nil)

And this test case no longer fails in Emacs 28 (but it fails in Emacs
27).  So is there more to be done in these merged bug reports?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#38912: 27.0.60; PDumper meets segmentation fault when evil is loaded

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> I noticed that byte-compiled functions don't hash consistently.  Here's
>> an ECM from Noam Postavsky
>> <https://github.com/skeeto/emacs-memoize/pull/10#issuecomment-415066682>:
>>
>>     (let ((obj1 (byte-compile (lambda (x) x)))
>>           (obj2 (byte-compile (lambda (x) x))))
>>       (list (equal obj1 obj2)
>>             (eq obj1 obj2)
>>             (= (sxhash obj1)
>>                (sxhash obj2))))
>>     ;=> (t nil nil)
>
> And this test case no longer fails in Emacs 28 (but it fails in Emacs
> 27).  So is there more to be done in these merged bug reports?

There wasn't any response in 43 weeks, so it seems like the issue has
been fixed for Emacs 28, and I'm therefore closing it.  If there are
other problems in this area, I think it would be better to open a new
bug report for those problems.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no