[ELPA] New package: emacs-gc-stats

[ELPA] New package: emacs-gc-stats

[Ihor Radchenko]

Hi,

I'd like to submit a helper package to collect GC statistics during
Emacs session. See the discussion in
https://yhetil.org/emacs-devel/831qluuj7e.fsf@gnu.org/

-----

This package collects Emacs garbage collection statistics over time and
saves it in the format that can be shared with Emacs maintainers.

*Usage:*

Add
#+begin_src emacs-lisp
(require 'emacs-gc-stats)
;; optional
(setq gc-cons-threshold
      (* 800000 (seq-random-elt '(1 2 4 8 16 32 64 128))))
(emacs-gc-stats-mode +1)
#+end_src
to your init file to enable the statistics acquiring.

When you are ready to share the results, run =M-x emacs-gc-stats-save-session=
and then share the saved ~emacs-gc-stats-file~ (defaults to
=~/.emacs.d/emacs-gc-stats.eld=).

You can use =M-x emacs-gc-stats-clear= to clear the currently collected
session data.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [ELPA] New package: emacs-gc-stats

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

> I'd like to submit a helper package to collect GC statistics during
> Emacs session. See the discussion in
> https://yhetil.org/emacs-devel/831qluuj7e.fsf@gnu.org/

Ping.

Re: [ELPA] New package: emacs-gc-stats

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Eli Zaretskii <eliz@gnu.org>
> Date: Fri, 09 Jun 2023 09:14:20 +0000
> 
> Ihor Radchenko <yantar92@posteo.net> writes:
> 
> > I'd like to submit a helper package to collect GC statistics during
> > Emacs session. See the discussion in
> > https://yhetil.org/emacs-devel/831qluuj7e.fsf@gnu.org/
> 
> Ping.

Stefan and Philip, please chime in.

Re: [ELPA] New package: emacs-gc-stats

[Stefan Monnier]

> I'd like to submit a helper package to collect GC statistics during
> Emacs session. See the discussion in
> https://yhetil.org/emacs-devel/831qluuj7e.fsf@gnu.org/

Let me know if you need help adding the package (I couldn't find a URL
for the code, so I have no opinion about it (yet :-))

I haven't read the above discussion but I'll just point out that
I regard discussions that focus on `gc-cons-threshold` to reflect
a misunderstanding.  What needs to be discussed/tweaked/investigated is
usually `gc-cons-percentage` instead.

More specifically the default value of `gc-cons-threshold` is rarely
used, since in most relevant circumstances `gc-cons-percentage` takes
precedence).


        Stefan


PS: In recent Emacsen, `gc-cons-percentage` is set to 1.0 when running in
    batch.  In non-batch I currently use 0.5 together with a timer that
    tries to opportunistically run the GC during idle time.

Re: [ELPA] New package: emacs-gc-stats

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I'd like to submit a helper package to collect GC statistics during
>> Emacs session. See the discussion in
>> https://yhetil.org/emacs-devel/831qluuj7e.fsf@gnu.org/
>
> Let me know if you need help adding the package (I couldn't find a URL
> for the code, so I have no opinion about it (yet :-))

Oops... https://git.sr.ht/~yantar92/emacs-gc-stats

This is my first time submitting to ELPA, so I just tried to follow ELPA
README.

> I haven't read the above discussion but I'll just point out that
> I regard discussions that focus on `gc-cons-threshold` to reflect
> a misunderstanding.  What needs to be discussed/tweaked/investigated is
> usually `gc-cons-percentage` instead.
>
> More specifically the default value of `gc-cons-threshold` is rarely
> used, since in most relevant circumstances `gc-cons-percentage` takes
> precedence).

I do not necessarily agree, but we should discuss it in the linked
thread, not here. The package just collects the statistics, including
`gc-cons-threshold', `gc-cons-percentage', and GC stats after every GC.

If the package gets added to ELPA, we can discuss further about what
kind of data we want to collect. Then, I can alter the README with
better instructions and go ahead asking Emacs users to record and later
report the data.

> PS: In recent Emacsen, `gc-cons-percentage` is set to 1.0 when running in
>     batch.  In non-batch I currently use 0.5 together with a timer that
>     tries to opportunistically run the GC during idle time.

This is yet another thing we can ask users to test, if we want to.
My idea is have users set GC settings randomly in their init file,
selecting from pre-configured set of alternatives we are yet to discuss.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [ELPA] New package: emacs-gc-stats

[Stefan Monnier]

>> Let me know if you need help adding the package (I couldn't find a URL
>> for the code, so I have no opinion about it (yet :-))
> Oops... https://git.sr.ht/~yantar92/emacs-gc-stats

Thanks, added.

>> PS: In recent Emacsen, `gc-cons-percentage` is set to 1.0 when running in
>>     batch.  In non-batch I currently use 0.5 together with a timer that
>>     tries to opportunistically run the GC during idle time.
> This is yet another thing we can ask users to test, if we want to.
> My idea is have users set GC settings randomly in their init file,
> selecting from pre-configured set of alternatives we are yet to discuss.

FWIW, see below the code I'm using.  On my main Emacs instance,
`gc--opportunistic-score` tells me:

    ((jit 17 0 0)
     (cmd 22 3 7)
     (earlier-gcs . 76)
     (opportunistic-gcs . 1478)
     (noncmd 194 6 52)
     (commands . 109401))

IOW the majority of GCs happen "opportunistically" (i.e. during
idle time).  The `cmd/noncmd` distinction is supposed to distinguish
those GCs that happen while running timers and process filters, but I'm
not sure my code is careful enough to keep track of those correctly.


        Stefan


;;;; Opportunistic GC

(defvar gc--opportunistic-last-gcs gcs-done)
(defvar gc--opportunistic-state 'noncmd)
(defvar gc--opportunistic-counters nil)

(defun gc--check ()
  (let ((gcs-counted
         (+ (alist-get 'multi-gcs gc--opportunistic-counters 0)
                   (alist-get 'earlier-gcs gc--opportunistic-counters 0)
                   (alist-get 'single-gc-cmds gc--opportunistic-counters 0)
                   (alist-get 'noncmds-gcs gc--opportunistic-counters 0)
                   (alist-get 'opportunistic-gcs gc--opportunistic-counters 0)
                   (or (car (alist-get 'error-gcs gc--opportunistic-counters)) 0))))
    (unless (= gcs-done gcs-counted)
      (push (+ (- gcs-done gcs-counted)
               (or (car (alist-get 'error-gcs gc--opportunistic-counters)) 0))
            (alist-get 'error-gcs gc--opportunistic-counters)))))

(defun gc--opportunistic-record (nextstate)
  (let ((counts (alist-get gc--opportunistic-state gc--opportunistic-counters)))
    (unless counts
      (setf (alist-get gc--opportunistic-state gc--opportunistic-counters)
            (setq counts (list 0 0 0))))
    (pcase (prog1 (- gcs-done gc--opportunistic-last-gcs)
             (setq gc--opportunistic-last-gcs gcs-done))
      ((pred (>= 0)) nil)
      (1 (cl-incf (nth 0 counts)))
      (gcs (cl-incf (nth 1 counts))
           (cl-incf (nth 2 counts) gcs))))
  (setq gc--opportunistic-state nextstate))

(defun gc--opportunistic-postch ()
  (cl-incf (alist-get 'commands gc--opportunistic-counters 0))
  (gc--opportunistic-record 'noncmd))

(defun gc--opportunistic-prech ()
  (cl-callf identity
      (alist-get 'earlier-gcs gc--opportunistic-counters gcs-done))
  (gc--opportunistic-record 'cmd)
  ;; (gc--check)
  )

(defun gc--opportunistic-jitlock (orig-fun start)
  (if (eq gc--opportunistic-state 'cmd)
      ;; Count jit-lock execution which happens during a command as
      ;; being part of command execution rather than as part of jit-lock!
      (funcall orig-fun start)
    (let ((gc--opportunistic-state gc--opportunistic-state))
      (gc--opportunistic-record 'jit)
      (unwind-protect
          (funcall orig-fun start)
        (gc--opportunistic-record 'postjit)))))

(add-hook 'pre-command-hook #'gc--opportunistic-prech)
(add-hook 'post-command-hook #'gc--opportunistic-postch)
(advice-add 'jit-lock-function :around #'gc--opportunistic-jitlock)

(defun gc--opportunistic ()
  "Run the GC during idle time."
  ;; This is good for two reasons:
  ;; - It reduces the number of times we have to GC in the middle of
  ;;   an operation.
  ;; - It means we GC when the C stack is short, reducing the risk of false
  ;;   positives from the conservative stack scanning.
  (unless (> gc-cons-percentage 0.1)
    (setq gc-cons-percentage 0.5))
  (when (garbage-collect-maybe 10)
    (cl-incf (alist-get 'opportunistic-gcs gc--opportunistic-counters 0))
    ;; Don't double count this GC in other categories.
    (cl-incf gc--opportunistic-last-gcs)
    ;; Recalibrate the timer.
    (cancel-function-timers #'gc--opportunistic)
    (run-with-idle-timer
     ;; FIXME: Magic formula!
     (+ 1 (* 10 (/ gc-elapsed gcs-done))) t #'gc--opportunistic)))


(defun gc--opportunistic-score ()
  "Show the current counters's that keep track of GC behavior."
  (interactive)
  (message "%S" gc--opportunistic-counters))

Re: [ELPA] New package: emacs-gc-stats

[Philip Kaludercic]

Ihor Radchenko <yantar92@posteo.net> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>>> Let me know if you need help adding the package (I couldn't find a URL
>>>> for the code, so I have no opinion about it (yet :-))
>>> Oops... https://git.sr.ht/~yantar92/emacs-gc-stats
>>
>> Thanks, added.
>
> For some reason, I am getting build failure:
>
> ======== Building tarball archive/emacs-gc-stats-1.0.tar...
> Problem with copyright notices:
> emacs-gc-stats/emacs-gc-stats.el:;; Copyright (C) 2023  Ihor Radchenko
> Missing copyright notice in emacs-gc-stats/emacs-gc-stats.el
> Build error for archive/emacs-gc-stats-1.0.tar: (error "Abort")
>
> Note that I have
>
>     ;; Copyright (C) 2023 Free Software Foundation, Inc.
>
> in <https://git.sr.ht/~yantar92/emacs-gc-stats/tree/main/item/emacs-gc-stats.el>
>
> What am I missing?

ELPA uses the latest commit that bumped the version tag in the package
header.  If that commit doesn't have the right copyright data, then that
is cause for the above error.

-- 
Philip Kaludercic

Re: [ELPA] New package: emacs-gc-stats

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Let me know if you need help adding the package (I couldn't find a URL
>>> for the code, so I have no opinion about it (yet :-))
>> Oops... https://git.sr.ht/~yantar92/emacs-gc-stats
>
> Thanks, added.

For some reason, I am getting build failure:

======== Building tarball archive/emacs-gc-stats-1.0.tar...
Problem with copyright notices:
emacs-gc-stats/emacs-gc-stats.el:;; Copyright (C) 2023  Ihor Radchenko
Missing copyright notice in emacs-gc-stats/emacs-gc-stats.el
Build error for archive/emacs-gc-stats-1.0.tar: (error "Abort")

Note that I have

    ;; Copyright (C) 2023 Free Software Foundation, Inc.

in <https://git.sr.ht/~yantar92/emacs-gc-stats/tree/main/item/emacs-gc-stats.el>

What am I missing?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [ELPA] New package: emacs-gc-stats

[Ihor Radchenko]

Philip Kaludercic <philipk@posteo.net> writes:

>> What am I missing?
>
> ELPA uses the latest commit that bumped the version tag in the package
> header.  If that commit doesn't have the right copyright data, then that
> is cause for the above error.

Ah. Makes sense.
Thanks for the clarification.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [ELPA] New package: emacs-gc-stats

[Stefan Monnier]

Ihor Radchenko [2023-06-11 09:15:31] wrote:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>>>> Let me know if you need help adding the package (I couldn't find a URL
>>>> for the code, so I have no opinion about it (yet :-))
>>> Oops... https://git.sr.ht/~yantar92/emacs-gc-stats
>>
>> Thanks, added.
>
> For some reason, I am getting build failure:
>
> ======== Building tarball archive/emacs-gc-stats-1.0.tar...
> Problem with copyright notices:
> emacs-gc-stats/emacs-gc-stats.el:;; Copyright (C) 2023  Ihor Radchenko
> Missing copyright notice in emacs-gc-stats/emacs-gc-stats.el
> Build error for archive/emacs-gc-stats-1.0.tar: (error "Abort")
>
> Note that I have
>
>     ;; Copyright (C) 2023 Free Software Foundation, Inc.
>
> in <https://git.sr.ht/~yantar92/emacs-gc-stats/tree/main/item/emacs-gc-stats.el>

But that's not the "version 1.0".

The tip has already built successfully (as can be seen in GNU-devel
ELPA: http://elpa.gnu.org/devel/emacs-gc-stats.html), so the next
version should presumably build fine in GNU ELPA when you release it.


        Stefan