bug#58338: 29.0.50; mapatoms called on more elements than in obarray?

bug#58338: 29.0.50; mapatoms called on more elements than in obarray?

[Philip Kaludercic]

From reading the docstring of `mapatoms', I would assume that this would
evaluate to t:

(let ((i 0))
  (mapatoms (lambda (_) (setq i (1+ i))) obarray)
  (= (length obarray) i))

But instead on my system I get (length obarray) ↝ 15121 and and i ↝
78050, which is are at a ratio of 5.16169565505.

Also interesting, if I intern a variable "foo", the value of i increases
to 83106, while (length obarray) stays the same (?)

Interestingly enough, I'd expect this to at least output a single
message:

(let ((ht (make-hash-table)))
  (mapatoms (lambda (s) (cl-incf (gethash s ht 0))))
  (maphash (lambda (sym i)
	     (when (> i 1)
	       (message "%S: %d" sym i)))
	   ht))

but apparently no symbol is invoked twice.  Not even by name

(let ((ht (make-hash-table :test #'equal)))
  (mapatoms (lambda (s) (cl-incf (gethash (symbol-name s) ht 0))))
  (maphash (lambda (sym i)
	     (when (> i 1)
	       (message "%S: %d" sym i)))
	   ht))

Is this perhaps related to symbol positions?  I don't quite understand
bare symbols yet, but it doesn't appear to change anything if I check

(let ((i 0))
  (mapatoms (lambda (s)
    (when (bare-symbol-p s)
      (setq i (1+ i))))
    obarray)
  (list (length obarray) i))

I am confused.  All I want is a max value for a progress reporter :(

In GNU Emacs 29.0.50 (build 14, x86_64-pc-linux-gnu, GTK+ Version
 3.24.34, cairo version 1.17.6) of 2022-10-06 built on rhea
Repository revision: 2c4c72b45d833b6ccdd2cc06ca3c6c9e9ad54357
Repository branch: feature/package+vc
System Description: Fedora Linux 36 (Workstation Edition)

Configured using:
 'configure --with-pgtk --with-imagemagick'

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ IMAGEMAGICK
JPEG JSON LIBSELINUX LIBSYSTEMD LIBXML2 MODULES NOTIFY INOTIFY PDUMPER
PGTK PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS XIM
GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  TeX-PDF-mode: t
  rcirc-color-mode: t
  rcirc-track-minor-mode: t
  global-git-commit-mode: t
  magit-auto-revert-mode: t
  outline-minor-mode: t
  flymake-mode: t
  yas-minor-mode: t
  flyspell-mode: t
  repeat-mode: t
  display-battery-mode: t
  display-time-mode: t
  diff-hl-flydiff-mode: t
  diff-hl-mode: t
  winner-mode: t
  windmove-mode: t
  corfu-history-mode: t
  corfu-mode: t
  vertico-multiform-mode: t
  vertico-mode: t
  electric-pair-mode: t
  shell-dirtrack-mode: t
  recentf-mode: t
  save-place-mode: t
  savehist-mode: t
  pixel-scroll-precision-mode: t
  pixel-scroll-mode: t
  xterm-mouse-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tab-bar-mode: t
  file-name-shadow-mode: t
  context-menu-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/home/philip/.config/emacs/site-lisp/emacs-inspector/inspector hides /home/philip/.config/emacs/elpa/inspector-0.8/inspector
/home/philip/.config/emacs/site-lisp/emacs-inspector/inspector-tests hides /home/philip/.config/emacs/elpa/inspector-0.8/inspector-tests
/home/philip/.config/emacs/site-lisp/flymake-proselint/flymake-proselint hides /home/philip/.config/emacs/elpa/flymake-proselint-0.3.0/flymake-proselint
/home/philip/.config/emacs/elpa/transient-0.3.7/transient hides /home/philip/Source/emacs/lisp/transient
/home/philip/.config/emacs/elpa/xref-1.5.1/xref hides /home/philip/Source/emacs/lisp/progmodes/xref

Features:
(shadow emacsbug descr-text solar cal-dst holidays holiday-loaddefs
cal-move loadhist reporter ert slime etags fileloop arc-mode
archive-mode hyperspec package-vc gnus-search eieio-opt speedbar ezimage
dframe flymake-proselint markdown-mode find-dired grep magit-ediff ediff
ediff-merg ediff-mult ediff-wind ediff-diff ediff-help ediff-init
ediff-util goto-addr eudc-capf eudc cus-start eudc-vars magit-patch
emacs-news-mode avy rect focus tex-info tex texmathp texinfo
texinfo-loaddefs gnus-fun sh-script smie executable shortdoc facemenu
two-column dictionary dictionary-connection ffap find-file magit-extras
face-remap pulse color xref ibuffer ibuffer-loaddefs cl-print
vertico-buffer consult-vertico consult compat-28 magit-bookmark bookmark
whitespace bug-reference mule-util flow-fill modus-vivendi-theme nndoc
gnus-dup url-cache display-line-numbers debbugs-gnu debbugs-compat
debbugs soap-client url-http url-auth url-gw rng-xsd rng-dt rng-util
xsd-regexp mailalias smtpmail autocrypt-message ecomplete rcirc-color
rcirc ietf-drums-date sort smiley gnus-cite mm-archive mail-extr textsec
uni-scripts idna-mapping ucs-normalize uni-confusable textsec-check qp
gnus-bcklg gnus-async gnus-ml disp-table vc-fossil vc-hg vc-bzr vc-src
vc-sccs vc-svn vc-cvs vc-rcs org-element avl-tree generator ol-eww eww
xdg url-queue mm-url ol-rmail ol-mhe ol-irc ol-info ol-gnus nnselect
ol-docview doc-view jka-compr image-mode exif ol-bibtex ol-bbdb ol-w3m
ol-doi org-link-doi org ob ob-tangle ob-ref ob-lob ob-table ob-exp
org-macro org-footnote org-src ob-comint org-pcomplete org-list
org-faces org-entities org-version ob-emacs-lisp ob-core ob-eval
org-table oc-basic bibtex ol org-keys oc org-compat advice org-macs
org-loaddefs cal-menu calendar cal-loaddefs char-fold misearch
multi-isearch dired-aux gnus-dired ef-duo-light-theme ef-duo-dark-theme
ef-night-theme ef-spring-theme ef-themes vc-git buffer-env compat
misterioso-theme cus-theme orderless vertico-directory vertico-flat
magit-submodule magit-obsolete magit-blame magit-stash magit-reflog
magit-bisect magit-push magit-pull magit-fetch magit-clone magit-remote
magit-commit magit-sequence magit-notes magit-worktree magit-tag
magit-merge magit-branch magit-reset magit-files magit-refs magit-status
magit magit-repos magit-apply magit-wip magit-log which-func imenu
edebug debug backtrace find-func magit-diff smerge-mode git-commit
log-edit add-log magit-core magit-autorevert autorevert filenotify
magit-margin magit-transient magit-process with-editor server magit-mode
transient edmacro kmacro magit-git magit-section magit-utils crm dash
vc-backup copyright autocrypt-gnus autocrypt nndraft nnmh utf-7 nnfolder
epa-file network-stream nsm gnus-agent gnus-srvr gnus-score score-mode
nnvirtual gnus-msg gnus-art mm-uu mml2015 mm-view mml-smime smime gnutls
dig nntp gnus-cache gnus-sum shr pixel-fill kinsoku url-file svg dom
gnus-group gnus-undo gnus-start gnus-dbus gnus-cloud nnimap nnmail
mail-source utf7 nnoo gnus-spec gnus-int gnus-range message yank-media
puny rfc822 mml mml-sec epa derived epg rfc6068 epg-config mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
gnus-win noutline outline checkdoc flymake-proc flymake warnings
yasnippet-snippets cl-extra yasnippet flyspell ispell auth-source-pass
repeat project battery dbus xml shell-command+ thingatpt dired-x dired
dired-loaddefs time sendmail rfc2047 rfc2045 ietf-drums gnus nnheader
gnus-util mail-utils range mm-util mail-prsvr finder-inf diff-hl-flydiff
diff diff-hl log-view pcvs-util vc-dir ewoc vc vc-dispatcher diff-mode
easy-mmode hippie-exp winner windmove corfu-history corfu
vertico-multiform vertico elec-pair tramp-cache time-stamp tramp-sh
tramp tramp-loaddefs trampver tramp-integration cus-edit pp icons
files-x tramp-compat rx shell pcomplete parse-time iso8601 time-date
ls-lisp format-spec recentf tree-widget wid-edit saveplace savehist
pixel-scroll cua-base xt-mouse modus-operandi-theme modus-themes pcase
cus-load setup site-lisp auto-site compile text-property-search comint
ansi-osc ansi-color autoload loaddefs-gen lisp-mnt debbugs-autoloads
rcirc-color-autoloads inspector-autoloads ef-themes-autoloads
flylisp-autoloads keycast-autoloads flymake-proselint-autoloads
which-key-autoloads geiser-guile-autoloads focus-autoloads
vertico-autoloads consult-autoloads vc-fossil-autoloads corfu-autoloads
diff-hl-autoloads crdt-autoloads embark-autoloads auctex-autoloads
tex-site magit-autoloads buffer-env-autoloads compat-autoloads
geiser-chibi-autoloads geiser-impl help-fns radix-tree help-mode
geiser-custom geiser-base ring geiser-autoloads slime-autoloads
transient-autoloads xref-autoloads info package browse-url url url-proxy
url-privacy url-expand url-methods url-history url-cookie
generate-lisp-file url-domsuf url-util mailcap url-handlers url-parse
auth-source cl-seq eieio eieio-core cl-macs password-cache json subr-x
map byte-opt gv bytecomp byte-compile cconv url-vars cl-loaddefs cl-lib
rmc iso-transl tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/pgtk-win pgtk-win term/common-win
pgtk-dnd tool-bar dnd fontset image regexp-opt fringe tabulated-list
replace newcomment text-mode lisp-mode prog-mode register page tab-bar
menu-bar rfn-eshadow isearch easymenu timer select scroll-bar mouse
jit-lock font-lock syntax font-core term/tty-colors frame minibuffer
nadvice seq simple cl-generic indonesian philippine cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite emoji-zwj charscript charprop
case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs faces cus-face macroexp files window
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget keymap hashtable-print-readable backquote threads dbusbind
inotify dynamic-setting system-font-setting font-render-setting cairo
gtk pgtk multi-tty make-network-process emacs)

Memory information:
((conses 16 1442594 267553)
 (symbols 48 63865 588)
 (strings 32 303656 13102)
 (string-bytes 1 10565718)
 (vectors 16 140881)
 (vector-slots 8 3010917 366454)
 (floats 8 3863 612)
 (intervals 56 72554 2447)
 (buffers 1000 114))

bug#58338: 29.0.50; mapatoms called on more elements than in obarray?

[Gerd Möllmann]

Philip Kaludercic <philipk@posteo.net> writes:

>>From reading the docstring of `mapatoms', I would assume that this would
> evaluate to t:
>
> (let ((i 0))
>   (mapatoms (lambda (_) (setq i (1+ i))) obarray)
>   (= (length obarray) i))
>
> But instead on my system I get (length obarray) ↝ 15121 and and i ↝
> 78050, which is are at a ratio of 5.16169565505.

obarray is a hash table. with each array element being a list of symbols
if not empty, for hash collisions.  And it's not a hash table of the
kind that make-hash-table makes, which I added much later.  And not a
Lisp list of symbols.

With your figures, it seems that there are on average 5 symbols per
bucket.  For details you'd have to look at the C code.

bug#58338: 29.0.50; mapatoms called on more elements than in obarray?

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Date: Thu, 06 Oct 2022 16:16:03 +0000
> 
> 
> >From reading the docstring of `mapatoms', I would assume that this would
> evaluate to t:
> 
> (let ((i 0))
>   (mapatoms (lambda (_) (setq i (1+ i))) obarray)
>   (= (length obarray) i))
> 
> But instead on my system I get (length obarray) ↝ 15121 and and i ↝
> 78050, which is are at a ratio of 5.16169565505.
> 
> Also interesting, if I intern a variable "foo", the value of i increases
> to 83106, while (length obarray) stays the same (?)

You assume that obarray is a flat vector or something?  Or how about
if you explain why the above surprised you?

bug#58338: 29.0.50; mapatoms called on more elements than in obarray?

[Stefan Kangas]

Philip Kaludercic <philipk@posteo.net> writes:

> From reading the docstring of `mapatoms', I would assume that this would
> evaluate to t:
>
> (let ((i 0))
>   (mapatoms (lambda (_) (setq i (1+ i))) obarray)
>   (= (length obarray) i))
>
> But instead on my system I get (length obarray) ↝ 15121 and and i ↝
> 78050, which is are at a ratio of 5.16169565505.
>
> Also interesting, if I intern a variable "foo", the value of i increases
> to 83106, while (length obarray) stays the same (?)

In my current long running Emacs session, I'm getting:

    15121 119188

Using emacs -Q on 29.0.50:

   15121 17228

Using emacs -Q on 27.1:

   15121 15880

> Interestingly enough, I'd expect this to at least output a single
> message:
>
> (let ((ht (make-hash-table)))
>   (mapatoms (lambda (s) (cl-incf (gethash s ht 0))))
>   (maphash (lambda (sym i)
> 	     (when (> i 1)
> 	       (message "%S: %d" sym i)))
> 	   ht))
>
> but apparently no symbol is invoked twice.  Not even by name
>
> (let ((ht (make-hash-table :test #'equal)))
>   (mapatoms (lambda (s) (cl-incf (gethash (symbol-name s) ht 0))))
>   (maphash (lambda (sym i)
> 	     (when (> i 1)
> 	       (message "%S: %d" sym i)))
> 	   ht))

Same results in Emacs 27.1.

> Is this perhaps related to symbol positions?  I don't quite understand
> bare symbols yet, but it doesn't appear to change anything if I check

It seems like this might be older than bare symbols.

bug#58338: 29.0.50; mapatoms called on more elements than in obarray?

[Philip Kaludercic]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>>>From reading the docstring of `mapatoms', I would assume that this would
>> evaluate to t:
>>
>> (let ((i 0))
>>   (mapatoms (lambda (_) (setq i (1+ i))) obarray)
>>   (= (length obarray) i))
>>
>> But instead on my system I get (length obarray) ↝ 15121 and and i ↝
>> 78050, which is are at a ratio of 5.16169565505.
>
> obarray is a hash table. with each array element being a list of symbols
> if not empty, for hash collisions.  And it's not a hash table of the
> kind that make-hash-table makes, which I added much later.  And not a
> Lisp list of symbols.

I see, from the value it appeared to be a flat vector.

> With your figures, it seems that there are on average 5 symbols per
> bucket.  For details you'd have to look at the C code.

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Date: Thu, 06 Oct 2022 16:16:03 +0000
>> 
>> 
>> >From reading the docstring of `mapatoms', I would assume that this would
>> evaluate to t:
>> 
>> (let ((i 0))
>>   (mapatoms (lambda (_) (setq i (1+ i))) obarray)
>>   (= (length obarray) i))
>> 
>> But instead on my system I get (length obarray) ↝ 15121 and and i ↝
>> 78050, which is are at a ratio of 5.16169565505.
>> 
>> Also interesting, if I intern a variable "foo", the value of i increases
>> to 83106, while (length obarray) stays the same (?)
>
> You assume that obarray is a flat vector or something?  Or how about
> if you explain why the above surprised you?

In that case what confuses me is that (length obarray) doesn't throw an
error.  Or what useful information does that give me?

bug#58338: 29.0.50; mapatoms called on more elements than in obarray?

[Gerd Möllmann]

On 22-10-06 20:21 , Philip Kaludercic wrote:

> In that case what confuses me is that (length obarray) doesn't throw an
> error.  Or what useful information does that give me?

It's a vector, and each vector has a length.

bug#58338: 29.0.50; mapatoms called on more elements than in obarray?

[Philip Kaludercic]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> On 22-10-06 20:21 , Philip Kaludercic wrote:
>
>> In that case what confuses me is that (length obarray) doesn't throw an
>> error.  Or what useful information does that give me?
>
> It's a vector, and each vector has a length.

But isn't the fact that it is a vector just a detail of the internal
implementation?

What is even more vexing is that the deceptively named `obarray-size'
just calls `length'.  Is there no better way to count the number of
entries in a obarray, as `hash-table-count' does for hash tables?

bug#58338: 29.0.50; mapatoms called on more elements than in obarray?

[Gerd Möllmann]

On 22-10-06 20:28 , Philip Kaludercic wrote:
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> 
>> On 22-10-06 20:21 , Philip Kaludercic wrote:
>>
>>> In that case what confuses me is that (length obarray) doesn't throw an
>>> error.  Or what useful information does that give me?
>>
>> It's a vector, and each vector has a length.
> 
> But isn't the fact that it is a vector just a detail of the internal
> implementation?

It's all history.  Obarray existed since the Big Bang.  Don't use it.

bug#58338: 29.0.50; mapatoms called on more elements than in obarray?

[Philip Kaludercic]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> On 22-10-06 20:28 , Philip Kaludercic wrote:
>> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>> 
>>> On 22-10-06 20:21 , Philip Kaludercic wrote:
>>>
>>>> In that case what confuses me is that (length obarray) doesn't throw an
>>>> error.  Or what useful information does that give me?
>>>
>>> It's a vector, and each vector has a length.
>> But isn't the fact that it is a vector just a detail of the internal
>> implementation?
>
> It's all history.  Obarray existed since the Big Bang.  Don't use it.

OK, then I'll close the report.  But if I am not supposed to use the
obarrays, what would you recommend if I want to count the number of
known symbols?  (Context:  I am trying to detect all variables defined
in a package, and since that takes a while I wrapped the `mapatoms' call in
a progress reporter, that needs a maximal value).

bug#58338: 29.0.50; mapatoms called on more elements than in obarray?

[Basil L. Contovounesios via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Philip Kaludercic [2022-10-06 18:51 +0000] wrote:

> OK, then I'll close the report.  But if I am not supposed to use the
> obarrays, what would you recommend if I want to count the number of
> known symbols?  (Context:  I am trying to detect all variables defined
> in a package, and since that takes a while I wrapped the `mapatoms' call in
> a progress reporter, that needs a maximal value).

An obarray is just a flat vector of symbols, but each symbol itself
is actually implemented in C as a singly linked list of symbols (see
struct Lisp_Symbol).  If you inspect mapatoms you'll see it indeed
traverses each symbol chain in each vector bucket.

> What is even more vexing is that the deceptively named `obarray-size'
> just calls `length'.  Is there no better way to count the number of
> entries in a obarray, as `hash-table-count' does for hash tables?

Thus I don't think it's possible to know ahead of time the exact number
of symbols interned in a given obarray without traversing the whole
structure with mapatoms (but maybe I'm forgetting something obvious).

For this and associated reasons hash tables are generally preferred over
obarrays as an associative data structure, but for variables/symbols
we're stuck with obarray.

HTH,

-- 
Basil

bug#58338: 29.0.50; mapatoms called on more elements than in obarray?

[Gerd Möllmann]

Philip Kaludercic <philipk@posteo.net> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>> It's all history.  Obarray existed since the Big Bang.  Don't use it.
>
> OK, then I'll close the report.  But if I am not supposed to use the
> obarrays, what would you recommend if I want to count the number of
> known symbols?

I guess you have to use mapatoms, then.

> (Context:  I am trying to detect all variables defined
> in a package,

Use do-symbols and alike.  Oh wait, that won't work, we don't have
packages.  Did I already mention that CL packages would be really nice
to have... :-).

> and since that takes a while I wrapped the `mapatoms' call in
> a progress reporter, that needs a maximal value).

You could also do it like Apple and Microsoft do: start displaying N%
with increasing N for the first minutes, until we reach 99%, and then
leave it at that until finished.  We could even offer that as a general
functionality, say with-apple-progress :-).

Or just display something that shows that it's still alive.  -, \, |, /,
for example.  Or the number of things processed so far.