bug#34404: 26.1; Finalizer in hash table run early?

bug#34404: 26.1; Finalizer in hash table run early?

[Braun Gábor]

Hi,

The following test is reported as failed in the *ert* buffer
but I think it should pass.

------------- File test.el ------------------
;; -*- lexical-binding: t; -*-
(require 'ert)

(ert-deftest finalizer-in-weak-hash-table ()
  "Test finalizers in weak hash tables"
  (skip-unless (fboundp 'make-finalizer))
  (let ((finalizer-run nil))
    (let ((table (make-hash-table :weakness 'key)))
      (let ((key (make-symbol "key")))
        ;; Without wrapper (ignore ...) the test passes, why?
        (ignore
         (puthash
          key
          (make-finalizer (lambda () (setq finalizer-run t)))
          table))
        (garbage-collect)
        ;; With just one (garbage-collect), the test passes.
        (garbage-collect)
        (message "Finalizer: %s" (gethash key table))
        (should-not finalizer-run)
        ;; Prevent optimizing out early.
        (message "Finalizer: %s" (gethash key table)))
      (garbage-collect)
      ;;(garbage-collect)
      (should finalizer-run))))
----------------------------------------------------------------

Start emacs with the command line:

emacs -Q -l test.el --eval '(ert t)'

In buffer *ert* press "m" with the cursor on the single test.
The following buffers are shown:

-------------- buffer *ert* ------------------------------------
Selector: t
Passed:  0
Failed:  1 (1 unexpected)
Skipped: 0
Total:   1/1

Started at:   2019-02-09 13:53:14+0100
Finished.
Finished at:  2019-02-09 13:53:15+0100

F

F finalizer-in-weak-hash-table
    Test finalizers in weak hash tables
    (ert-test-failed
     ((should-not finalizer-run)
      :form finalizer-run :value t))


----------------------------------------------------------------


------------------ buffer *ERT Messages* -----------------------
Messages for test ‘finalizer-in-weak-hash-table’:
Finalizer: #<finalizer used>
----------------------------------------------------------------

Discussion:

The finalizer since its creation is always reachable in the inner-most 
(let...)
via (gethash key table), so it shouldn't be run until the last
(message...) there, which accesses the finalizer this way.
However, both the message made in buffer *ERT Messages* as well as the 
failed
form in buffer *ert* indicates that the finalizer has been run; that is
the only way the local variable finalizer-run can have a value t.

(The (message...) after the failed test (should-not...) is there to rule
out the destruction of local variables key and table early as a form of
optimization, which would justify running the finalizer before the 
test.)

As some of the comments in the test indicate,
small changes cause the test to pass:
like omitting the wrapper (ignore...) around (puthash...)
or just one (garbage-collection) before (should-not...) causes the test
to pass.

System information:

In GNU Emacs 26.1 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.24.2)
 of 2018-12-26, modified by Debian built on x86-ubc-01
Windowing system distributor 'The X.Org Foundation', version 
11.0.12003000
System Description:	Debian GNU/Linux buster/sid

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Finalizer: #<finalizer used>
Ran 1 tests, 0 results were as expected, 1 unexpected
Making completion list...

Configured using:
 'configure --build x86_64-linux-gnu --prefix=/usr
 --sharedstatedir=/var/lib --libexecdir=/usr/lib
 --localstatedir=/var/lib --infodir=/usr/share/info
 --mandir=/usr/share/man --enable-libsystemd --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/26.1/site-
lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/26.1/site-lisp:/
usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils --build
 x86_64-linux-gnu --prefix=/usr --sharedstatedir=/var/lib
 --libexecdir=/usr/lib --localstatedir=/var/lib
 --infodir=/usr/share/info --mandir=/usr/share/man --enable-libsystemd
 --with-pop=yes
 --enable-locallisppath=/etc/emacs:/usr/local/share/emacs/26.1/site-
lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/26.1/site-lisp:/
usr/share/emacs/site-lisp
 --with-sound=alsa --without-gconf --with-mailutils --with-x=yes
 --with-x-toolkit=gtk3 --with-toolkit-scroll-bars 'CFLAGS=-g -O2
 -fdebug-prefix-map=/build/emacs-3ThesY/emacs-26.1+1=. -fstack-
protector-strong
 -Wformat -Werror=format-security -Wall' 'CPPFLAGS=-Wdate-time
 -D_FORTIFY_SOURCE=2' LDFLAGS=-Wl,-z,relro'

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS NOTIFY
ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 THREADS LIBSYSTEMD LCMS2

Important settings:
  value of $LANG: hu_HU.UTF-8
  locale-coding-system: utf-8-unix

Major mode: ERT-View

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny seq byte-opt bytecomp
byte-compile cconv dired dired-loaddefs format-spec rfc822 mml mml-sec
password-cache epa derived epg epg-config gnus-util rmail rmail-loaddefs
mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils
mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
mail-utils cl-seq thingatpt cl-extra help-mode cl-macs gv ert pp
find-func ewoc easymenu debug cl-loaddefs cl-lib elec-pair time-date
mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel term/x-win x-win term/common-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list replace newcomment
text-mode elisp-mode lisp-mode prog-mode register page menu-bar
rfn-eshadow isearch timer select scroll-bar mouse jit-lock font-lock
syntax facemenu font-core term/tty-colors frame cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote dbusbind inotify lcms2
dynamic-setting system-font-setting font-render-setting move-toolbar gtk
x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 107828 8705)
 (symbols 48 21253 1)
 (miscs 40 63 154)
 (strings 32 30662 1206)
 (string-bytes 1 809252)
 (vectors 16 16338)
 (vector-slots 8 511222 11260)
 (floats 8 60 178)
 (intervals 56 366 8)
 (buffers 992 14))


Best wishes,

     Gábor Braun

bug#34404: 26.1; Finalizer in hash table run early?

[Lars Ingebrigtsen]

Braun Gábor <braungb88@gmail.com> writes:

> As some of the comments in the test indicate,
> small changes cause the test to pass:
> like omitting the wrapper (ignore...) around (puthash...)
> or just one (garbage-collection) before (should-not...) causes the test
> to pass.

This does seem like a bug...  if Emacs is using a mark-and-sweep garbage
collector, which I thought it did.  But Andreas (in a different bug
report) seemed to imply that Emacs uses a conservative garbage collector
these days, and I see that there's some talk about that in the source
code, but...  it's documented to be a mark-and-sweep gc.

So now I'm confused: Is Emacs still using a pure mark-and-sweep gc as
the Emacs Internals section of the elisp manual says?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#34404: 26.1; Finalizer in hash table run early?

[Andreas Schwab]

On Nov 26 2020, Lars Ingebrigtsen wrote:

> This does seem like a bug...  if Emacs is using a mark-and-sweep garbage
> collector, which I thought it did.  But Andreas (in a different bug
> report) seemed to imply that Emacs uses a conservative garbage collector
> these days, and I see that there's some talk about that in the source
> code, but...  it's documented to be a mark-and-sweep gc.

There's no contradiction, the concepts are orthogonal.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#34404: 26.1; Finalizer in hash table run early?

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Thu, 26 Nov 2020 13:15:45 +0100
> Cc: 34404@debbugs.gnu.org
> 
> This does seem like a bug...  if Emacs is using a mark-and-sweep garbage
> collector, which I thought it did.  But Andreas (in a different bug
> report) seemed to imply that Emacs uses a conservative garbage collector
> these days, and I see that there's some talk about that in the source
> code, but...  it's documented to be a mark-and-sweep gc.
> 
> So now I'm confused: Is Emacs still using a pure mark-and-sweep gc as
> the Emacs Internals section of the elisp manual says?

Yes.

bug#34404: 26.1; Finalizer in hash table run early?

[martin rudalics]

 > This does seem like a bug...  if Emacs is using a mark-and-sweep garbage
 > collector, which I thought it did.  But Andreas (in a different bug
 > report) seemed to imply that Emacs uses a conservative garbage collector
 > these days, and I see that there's some talk about that in the source
 > code, but...  it's documented to be a mark-and-sweep gc.
 >
 > So now I'm confused: Is Emacs still using a pure mark-and-sweep gc as
 > the Emacs Internals section of the elisp manual says?

Conservative implies mark and sweep.  You can't relocate objects in the
heap because that would invalidate any pointers from the stack.

martin

bug#34404: 26.1; Finalizer in hash table run early?

[Lars Ingebrigtsen]

martin rudalics <rudalics@gmx.at> writes:

>> So now I'm confused: Is Emacs still using a pure mark-and-sweep gc as
>> the Emacs Internals section of the elisp manual says?
>
> Conservative implies mark and sweep.  You can't relocate objects in the
> heap because that would invalidate any pointers from the stack.

But mark and sweep does not imply conservative -- you can have a mark
and sweep gc that will garbage-collect 100% of the garbage, which a
conservative gc typically will not.

The Emacs documentation seems to imply that Emacs has one of these 100%
garbage collectors, which is what this bug report is about, I think: The
user has an object that the gc should collect, but it's not collected
(always).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#34404: 26.1; Finalizer in hash table run early?

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Fri, 27 Nov 2020 08:56:33 +0100
> Cc: 34404@debbugs.gnu.org, Braun Gábor <braungb88@gmail.com>
> 
> The Emacs documentation seems to imply that Emacs has one of these 100%
> garbage collectors, which is what this bug report is about, I think: The
> user has an object that the gc should collect, but it's not collected
> (always).

The question is how do you define "100%"?  The Emacs GC is
conservative in the sense that when there's a doubt whether an object
_could_ be still referenced, we mark it so it doesn't get swept.

bug#34404: 26.1; Finalizer in hash table run early?

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> The question is how do you define "100%"?  The Emacs GC is
> conservative in the sense that when there's a doubt whether an object
> _could_ be still referenced, we mark it so it doesn't get swept.

When is Emacs in doubt?  :-)  If we could say something about this in
the manual (or the `garbage-collect' doc string), I think that'd be
helpful -- people do seem to think that calling `garbage-collect' is
guaranteed to collect all the garbage.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#34404: 26.1; Finalizer in hash table run early?

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: rudalics@gmx.at,  34404@debbugs.gnu.org,  braungb88@gmail.com
> Date: Fri, 27 Nov 2020 09:33:15 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The question is how do you define "100%"?  The Emacs GC is
> > conservative in the sense that when there's a doubt whether an object
> > _could_ be still referenced, we mark it so it doesn't get swept.
> 
> When is Emacs in doubt?  :-)

Mainly when we find what looks like Lisp objects on the C stack.

> If we could say something about this in the manual (or the
> `garbage-collect' doc string), I think that'd be helpful -- people
> do seem to think that calling `garbage-collect' is guaranteed to
> collect all the garbage.

Fine by me, please feel free to suggest such wording.

bug#34404: 26.1; Finalizer in hash table run early?

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> If we could say something about this in the manual (or the
>> `garbage-collect' doc string), I think that'd be helpful -- people
>> do seem to think that calling `garbage-collect' is guaranteed to
>> collect all the garbage.
>
> Fine by me, please feel free to suggest such wording.

OK, I've taken a stab at it on master, and I'm closing this bug report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no