bug#6069: 23.1.96; sxhash, big uints and overflow-error in custom-file

bug#6069: 23.1.96; sxhash, big uints and overflow-error in custom-file

[David Reitter]

Not sure if this is an error of Emacs, or if the fault is mine:

- `sxhash' of some object produced a 32-bit unsigned integer, which was represented just fine in Lisp.

- I stored it in a customization variable, which was subsequently saved to my custom-file.

- When reading back the file, I got an overflow error, as in

(read-from-string "2475893479")
--> overflow error


Due to the nature of hashes, it appears unreasonable hard for me to make a test case.  But looking at the code,  sxhash is designed to produce an unsigned int and make a (signed?) Lisp integer from it.

 INTMASK is 0xffffffff at least on my system.

Artihmetic operations, OTOH convert big ints to floats in arith_driver():

(let ((bigint (+ 1475893479 1000000000)))
  (floatp bigint))


(Would sxhash in that case produce a negative value if its result is printed? I don't know.
Somewhere along the way to `custom-save-variables', some info about the meaning of bit #31 gets lost, and Emacs produces string representation that can't be read.)




In GNU Emacs 23.1.96.1 (i386-apple-darwin9.8.0, NS apple-appkit-949.54)
of 2010-04-30 on braeburn.aquamacs.org - Aquamacs Distribution 2.0preview6
Windowing system distributor `Apple', version 10.3.1038
configured using `configure  '--with-ns' '--without-x' 'CFLAGS=-arch i386 -arch ppc' 'LDFLAGS=-arch i386 -arch ppc''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Text

Minor modes in effect:
  which-function-mode: t
  savehist-mode: t
  smart-frame-positioning-mode: t
  aquamacs-autoface-mode: t
  recentf-mode: t
  osx-key-mode: t
  tabbar-mwheel-mode: t
  tabbar-mode: t
  show-paren-mode: t
  delete-selection-mode: t
  pc-selection-mode: t
  cua-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  smart-spacing-mode: t
  column-number-mode: t
  line-number-mode: t
  visual-line-mode: t
  transient-mark-mode: t

Recent input:
E G A T I V E SPC V B A L U <backspace> <backspace> 
<backspace> <backspace> A L U E ? <up> <return> <down-mouse-1> 
<mouse-movement> <mouse-1> <return> ⇧D O E SPC S <backspace> 
<backspace> S SPC T H E R SPC ` P 0 R I N <backspace> 
<backspace> <backspace> <backspace> <backspace> <backspace> 
<backspace> <backspace> SPC <backspace> <backspace> 
<backspace> <backspace> <switch-frame> ^H V ^G ^H F 
C U S T O M I <backspace> - S A V E - V <tab> <return> 
<help-echo> <help-echo> <down-mouse-1> <mouse-2> <help-echo> 
<help-echo> <wheel-down> <double-wheel-down> <triple-wheel-down> 
<triple-wheel-down> <triple-wheel-down> <triple-wheel-down> 
<wheel-down> <double-wheel-down> <triple-wheel-down> 
<triple-wheel-down> <wheel-up> <double-wheel-up> <triple-wheel-up> 
<triple-wheel-up> <triple-wheel-up> <wheel-down> <double-wheel-down> 
<triple-wheel-down> <triple-wheel-down> <triple-wheel-down> 
<triple-wheel-down> <triple-wheel-down> <wheel-down> 
<double-wheel-down> <triple-wheel-down> <wheel-up> 
<double-wheel-up> <triple-wheel-up> <triple-wheel-up> 
<triple-wheel-up> <triple-wheel-up> <triple-wheel-up> 
<triple-wheel-up> <triple-wheel-up> <triple-wheel-up> 
<wheel-down> <double-wheel-down> <triple-wheel-down> 
<triple-wheel-down> <wheel-down> <double-wheel-down> 
<down-mouse-1> <mouse-1> <double-down-mouse-1> <double-mouse-1> 
⌘C <switch-frame> <down-mouse-1> <mouse-movement> <mouse-movement> 
<mouse-movement> <mouse-movement> <drag-mouse-1> ⇧S 
O M E W H E R E SPC A L O N M G <backspace> <backspace> 
G SPC T H E SPC W A Y SPC T O SPC ` ⌘V ' SPC S <backspace> 
<backspace> M <backspace> , SPC S O M E T <backspace> 
<backspace> <backspace> <backspace> <backspace> T H 
I S SPC G E T S SPC L O S T SPC <backspace> , SPC A 
N D SPC I T SPC P R O D U C E S SPC A SPC C U S T O 
<backspace> <backspace> <backspace> <backspace> <backspace> 
<backspace> <backspace> S T R I N G SPC R E P R E S 
E N T A T I O N SPC T H A T SPC C A N ' T SPC B E SPC 
R E A D SPC B <backspace> <backspace> . <return> <return> 
<ns-application-activated> ⌘A ⌘C <switch-frame> <ns-application-activated> 
<switch-frame> <menu-bar> <help-menu> <bug-diagnosis> 
<send-emacs-bug-report>

Recent messages:
Making completion list...
Type "q" to quit.
if: Search failed: "arith_d"
Mark set
Undo...
Undo!
Mark set
Quit
uncompressing cus-edit.el.gz...done
Unable to load color "dark cyan"

Load-path shadows:
/Users/dr/Library/Preferences/Aquamacs Emacs/Recent Files hides /Users/dr/Library/Preferences/Aquamacs Emacs/Aquamacs Emacs2/Recent Files
/Users/dr/Library/Preferences/Aquamacs Emacs/Preferences hides /Users/dr/Library/Preferences/Aquamacs Emacs/Aquamacs Emacs2/Preferences
/Users/dr/Library/Preferences/Aquamacs Emacs/places hides /Users/dr/Library/Preferences/Aquamacs Emacs/Aquamacs Emacs2/places
/Users/dr/Library/Preferences/Aquamacs Emacs/minibuffer-history hides /Users/dr/Library/Preferences/Aquamacs Emacs/Aquamacs Emacs2/minibuffer-history
/Users/dr/Library/Preferences/Aquamacs Emacs/frame-positions hides /Users/dr/Library/Preferences/Aquamacs Emacs/Aquamacs Emacs2/frame-positions
/Users/dr/Library/Preferences/Aquamacs Emacs/customizations hides /Users/dr/Library/Preferences/Aquamacs Emacs/Aquamacs Emacs2/customizations
/Users/dr/Library/Application Support/Aquamacs Emacs/JDEE/site-start hides /Library/Application Support/Aquamacs Emacs/SLIME/site-start
/Users/dr/Library/Application Support/Aquamacs Emacs/JDEE/site-start hides /Users/dr/Downloads/Aquamacs 2.app/Contents/Resources/lisp/aquamacs/site-start

Features:
(shadow sort mail-extr message ecomplete rfc822 mml mml-sec
password-cache mm-decode mm-bodies mm-encode mailcap mail-parse rfc2231
rfc2047 rfc2045 qp ietf-drums mailabbrev nnheader gnus-util netrc
mm-util mail-prsvr gmm-utils mailheader canlock sha1 hex-util hashcash
mail-utils emacsbug jka-compr multi-isearch vc-git cc-mode cc-fonts
cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs find-func
pp which-func imenu slime-autoloads load-emacs-plugins
aquamacs-mode-defaults auctex-config server tex-site smart-dnd
aquamacs-aux savehist mouse-sel one-buffer-one-frame
smart-frame-positioning drews_init color-theme-autoloads saveplace
visual-line aquamacs-bug aquamacs-autoface-mode aquamacs-editing
sendmail recentf tree-widget cus-edit osxkeys emulate-mac-keyboard-mode
frame-cmds strings misc-fns thingatpt+ thingatpt frame-fns avoid
aquamacs-mac-fontsets fit-frame aquamacs-frame-setup aquamacs-tabbar
tabbar-window cl cl-19 tabbar easy-mmode cus-start cus-load
load-emacs-pre-plugins aquamacs-site-start cocoa-compatibility filladapt
aquamacs-redo check-for-updates aquamacs-menu osx_defaults
aquamacs-tool-bar aquamacs mac-extra-functions aquamacs-tools
aquamacs-macros parse-time timezone time-date paren delsel pc-select
cua-base wid-edit regexp-opt advice advice-preload byte-opt bytecomp
byte-compile debug help-fns help-mode view image-file disp-table tooltip
ediff-hook vc-hooks lisp-float-type mwheel ns-win easymenu tool-bar dnd
fontset image fringe lisp-mode register page menu-bar rfn-eshadow timer
select scroll-bar mldrag mouse jit-lock font-lock syntax facemenu
font-core frame cham georgian utf-8-lang misc-lang vietnamese tibetan
thai tai-viet lao korean japanese hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese case-table epa-hook
jka-cmpr-hook help simple abbrev loaddefs button minibuffer faces
cus-face files text-properties overlay md5 base64 format env code-pages
mule custom widget hashtable-print-readable backquote
make-network-process ns multi-tty emacs)

Command line: (/Users/dr/Downloads/Aquamacs 2.app/Contents/MacOS/Aquamacs)

PATH: /opt/local/bin:/opt/local/sbin:/opt/local/bin:/opt/local/sbin:/opt/local/bin:/opt/local/sbin:/Library/Frameworks/Python.framework/Versions/Current/bin:/opt/local/lib/postgresql82/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/git/bin:/usr/texbin:/usr/X11/bin:/Users/dr/Applications:/Users/dr/Applications/bin:/Users/dr/Projects/openccg/bin:/usr/texbin/powerpc-apple-darwin-current:/usr/local/git/bin:/Developer/Platforms/iPhoneFOSS.platform/Developer/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/texbin:/usr/local/texlive/2009/bin

exec-path: (/opt/local/bin /opt/local/sbin /opt/local/bin /opt/local/sbin /opt/local/bin /opt/local/sbin /Library/Frameworks/Python.framework/Versions/Current/bin /opt/local/lib/postgresql82/bin /opt/local/bin /opt/local/sbin /usr/bin /bin /usr/sbin /sbin /usr/local/bin /usr/local/git/bin /usr/texbin /usr/X11/bin /Users/dr/Applications /Users/dr/Applications/bin /Users/dr/Projects/openccg/bin /usr/texbin/powerpc-apple-darwin-current /usr/local/git/bin /Developer/Platforms/iPhoneFOSS.platform/Developer/bin /usr/bin /bin /usr/sbin /sbin /Users/dr/Downloads/Aquamacs 2.app/Contents/MacOS/bin /usr/local/texlive/2009/bin)

bug#6069: INTMASK on 32/64 bit machines

[David Reitter]

I have one post-scriptum.

I just noticed that the large `sxhash' integer was produced on a 64-bit machine with INTMASK 0xffffffff,
but that the read failure occurred with a binary compiled for 32-bit architectures.  In that Emacs, INTMASK is 0x3fffffff.

I think that explains what I'm seeing.

Would it be sensible to make `sxhash' use a lower common denominator for modern machines, such as 0x3fffffff?

Or, should the lower INTMASK be used on 64-bit architectures as well?

It's less than ideal that printed Lisp expressions, especially those in customization files, are not interchangeable between different builds of the same version of Emacs.

bug#6069: 23.1.96; sxhash, big uints and overflow-error in custom-file

[Stefan Monnier]

> - I stored it in a customization variable, which was subsequently
>   saved to my custom-file.

What was the use of this sxhash value in your customization?


        Stefan

bug#6069: 23.1.96; sxhash, big uints and overflow-error in custom-file

[David Reitter]

On Mar 27, 2011, at 4:46 PM, Stefan Monnier wrote:

>> - I stored it in a customization variable, which was subsequently
>>  saved to my custom-file.
> 
> What was the use of this sxhash value in your customization?

I allow users to customize the toolbar through a standard graphical interface.  This customization is specific to the toolbar set by the major mode (and other contextual factors).  Thus, I associate a toolbar setting with a hash over the keys used in the tool-bar-map.  This is not ideal (as other keymaps may also define icons that appear in the toolbar), but works reasonably well. 

- D

bug#6069: 23.1.96; sxhash, big uints and overflow-error in custom-file

[Stefan Monnier]

>>> - I stored it in a customization variable, which was subsequently
>>> saved to my custom-file.
>> What was the use of this sxhash value in your customization?
> I allow users to customize the toolbar through a standard graphical
> interface.  This customization is specific to the toolbar set by the major
> mode (and other contextual factors).  Thus, I associate a toolbar setting
> with a hash over the keys used in the tool-bar-map.  This is not ideal (as
> other keymaps may also define icons that appear in the toolbar), but works
> reasonably well. 

So you could work around the problem by doing a logand with #xfffffff.
I'm not saying that the problem you describe is an Emacs feature, but
simply that the only way to "make it work right" (i.e. make that sxhash
returns the same value on all systems) involves clipping it to #xfffffff
and while that's OK in many cases, I'd rather do it when it can
be avoided (e.g. when the sxhash value is used to index a hash table).

Also sxhash's values are not guaranteed to be the same between different
Emacs versions, so maybe instead of storing the sxhash you should simply
use the list of keys as-is.  It's not like it's expected to be very long.


        Stefan

bug#6069: 23.1.96; sxhash, big uints and overflow-error in custom-file

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Also sxhash's values are not guaranteed to be the same between different
> Emacs versions, so maybe instead of storing the sxhash you should simply
> use the list of keys as-is.  It's not like it's expected to be very long.

This doesn't seem to be a bug -- the sxhash values are system-dependent
and shouldn't be used directly when storing, so I'm closing this bug
report.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/