bug#30943: save-hist creates massive cache file

bug#30943: save-hist creates massive cache file

[Chris Findeisen]

[-- Attachment #1: Type: text/plain, Size: 879 bytes --]

save-hist-additional-variables never get truncated by save-hist, leading
to a massive cache file and slowdown. Practically, this matters when the
cache file silently grows to 1/2 a GB, and emacs begins randomly freezing.

history-length is supposed to keep a limit on the max history for
save-hist-additional-variables, but it doesn't.

In the savehist-save function:

(dolist (symbol savehist-minibuffer-history-variables)
          (when (and (boundp symbol)
                     (not (memq symbol savehist-ignored-variables)))
            (let ((value (savehist-trim-history (symbol-value symbol)))
      ;;....
  ))))

(dolist (symbol savehist-additional-variables)
  (when (boundp symbol)
    (let ((value (symbol-value symbol)))
    (when (savehist-printable value)
    ;; ...
    ))))

As you can see, the save-hist-trim-history fn is not called in the
second code block.

[-- Attachment #2: Type: text/html, Size: 16919 bytes --]

bug#30943: save-hist creates massive cache file

[Eli Zaretskii]

> From: Chris Findeisen <cfindeisen@google.com>
> Date: Sun, 25 Mar 2018 19:11:21 +0000
> 
> save-hist-additional-variables never get truncated by save-hist, leading
> to a massive cache file and slowdown. Practically, this matters when the cache file silently grows to 1/2 a
> GB, and emacs begins randomly freezing.
> 
> history-length is supposed to keep a limit on the max history for save-hist-additional-variables, but it doesn't.
> 
> In the savehist-save function:
> 
> (dolist (symbol savehist-minibuffer-history-variables)
>           (when (and (boundp symbol)
>                      (not (memq symbol savehist-ignored-variables)))
>             (let ((value (savehist-trim-history (symbol-value symbol)))
>       ;;....
>   ))))
> 
> (dolist (symbol savehist-additional-variables)
>   (when (boundp symbol)
>     (let ((value (symbol-value symbol)))
>     (when (savehist-printable value)
>     ;; ...
>     ))))
> 
> As you can see, the save-hist-trim-history fn is not called in the
> second code block.

I'm not sure I understand the situation.  Is the variable you are
trying to save using save-hist a long list or something, and it keeps
growing from session to session without any limitation?

Save-hist doesn't truncate any variables except the ones it knows
about, because it cannot be sure that truncating some list might leave
the variable with an invalid value.  For variables like
minibuffer-history, it knows that truncating the history will produce
a history that is still valid, but how can it do the same with other
variables?  For example, it could be that some other variable needs to
be truncated from the other end of the list, otherwise the value will
be useless.

If I'm missing something about your use case, please tell the details,
and perhaps show an example of a variable with which this happens.

Thanks.

bug#30943: save-hist creates massive cache file

[Chris Findeisen]

[-- Attachment #1: Type: text/plain, Size: 3864 bytes --]

Hey Eli,

Correct, the issue is that the variable is growing from session to session
without limitation.

For instance, some of the culprits from my particular configuration:
grep -E -b -o '^\(setq [^ ]+' ~/.emacs.d/.cache/savehist
....
51869:(setq evil-jumps-history
53265:(setq mark-ring
53287:(setq search-ring
53311:(setq regexp-search-ring
53579:(setq extended-command-history

I've also seen online this issue crop up for others:
https://emacs.stackexchange.com/questions/12086/abnormally-large-savehist-file

Save-hist doesn't truncate any variables except the ones it knows
> about, because it cannot be sure that truncating some list might leave
> the variable with an invalid value.  For variables like
> minibuffer-history, it knows that truncating the history will produce
> a history that is still valid, but how can it do the same with other

variables?

For example, it could be that some other variable needs to
> be truncated from the other end of the list, otherwise the value will
> be useless.


Hmm, I understand the dilemma about not knowing the format, however there
ought to be an alternative way of adding variables that we can "inform"
save-hist about. Whether that's demanding/documenting a valid format, "Hey
if you add this variable to be tracked by save-hist, it must be formatted
like so" or providing a secondary argument that takes a function specifying
custom handling, I'm not sure.

I think it makes sense to provide some method of honoring the
history-length variable. If the list is never culled, any emacs session
will just grow over time. That is almost never desired(at least for me),
and I would see no point to have additional-variables to be track (I would
have to worry about my buffer silently slowing me down).

Let me know if I'm missing something, or if you have alternate ideas.
Thanks!

Regards,
Chris Findeisen

On Mon, Mar 26, 2018 at 7:59 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Chris Findeisen <cfindeisen@google.com>
> > Date: Sun, 25 Mar 2018 19:11:21 +0000
> >
> > save-hist-additional-variables never get truncated by save-hist, leading
> > to a massive cache file and slowdown. Practically, this matters when the
> cache file silently grows to 1/2 a
> > GB, and emacs begins randomly freezing.
> >
> > history-length is supposed to keep a limit on the max history for
> save-hist-additional-variables, but it doesn't.
> >
> > In the savehist-save function:
> >
> > (dolist (symbol savehist-minibuffer-history-variables)
> >           (when (and (boundp symbol)
> >                      (not (memq symbol savehist-ignored-variables)))
> >             (let ((value (savehist-trim-history (symbol-value symbol)))
> >       ;;....
> >   ))))
> >
> > (dolist (symbol savehist-additional-variables)
> >   (when (boundp symbol)
> >     (let ((value (symbol-value symbol)))
> >     (when (savehist-printable value)
> >     ;; ...
> >     ))))
> >
> > As you can see, the save-hist-trim-history fn is not called in the
> > second code block.
>
> I'm not sure I understand the situation.  Is the variable you are
> trying to save using save-hist a long list or something, and it keeps
> growing from session to session without any limitation?
>
> Save-hist doesn't truncate any variables except the ones it knows
> about, because it cannot be sure that truncating some list might leave
> the variable with an invalid value.  For variables like
> minibuffer-history, it knows that truncating the history will produce
> a history that is still valid, but how can it do the same with other
> variables?  For example, it could be that some other variable needs to
> be truncated from the other end of the list, otherwise the value will
> be useless.
>
> If I'm missing something about your use case, please tell the details,
> and perhaps show an example of a variable with which this happens.
>
> Thanks.
>

[-- Attachment #2: Type: text/html, Size: 9723 bytes --]

bug#30943: save-hist creates massive cache file

[Glenn Morris]

Chris Findeisen wrote:

> grep -E -b -o '^\(setq [^ ]+' ~/.emacs.d/.cache/savehist
> ....
> 51869:(setq evil-jumps-history
> 53265:(setq mark-ring
> 53287:(setq search-ring
> 53311:(setq regexp-search-ring
> 53579:(setq extended-command-history

None of those look large.
You either want to post the whole output, or find the one where the
offset jumps significantly from the previous match.

bug#30943: save-hist creates massive cache file

[Chris Findeisen]

[-- Attachment #1: Type: text/plain, Size: 1523 bytes --]

>
> None of those look large.


yeah, I had rm'd my cache file when it was really huge (~1GB), now it's
regrowing. To get a sense of what I was looking at previously, I had a
similar output to this person's
https://emacs.stackexchange.com/questions/12086/abnormally-large-savehist-file.
The point is that over time these variables grow and grow and slow.

savehist doesn't truncate anything AFAICS.
> It just assumes variables are trimmed by add-to-history etc in the
> course of normal operation. So if you add a non-history variable to
> savehist (or set history-length to t), it will eventually blow up.
>

Hmm, ok. I may have misunderstood savehist's role then, I thought that
savehist managed history size. Eli, apparently the savehist-trim-history
function is only used on XEmacs--it's a noop on Emacs.

Sorry for the confusion then. First bug report :D ... failed bug report :(
. I'll talk to Spacemacs folks about fixing their configuration to guard
these variables accordingly. Thanks for your time!

Regards,
Chris Findeisen


On Tue, Mar 27, 2018 at 12:44 PM Glenn Morris <rgm@gnu.org> wrote:

> Chris Findeisen wrote:
>
> > grep -E -b -o '^\(setq [^ ]+' ~/.emacs.d/.cache/savehist
> > ....
> > 51869:(setq evil-jumps-history
> > 53265:(setq mark-ring
> > 53287:(setq search-ring
> > 53311:(setq regexp-search-ring
> > 53579:(setq extended-command-history
>
> None of those look large.
> You either want to post the whole output, or find the one where the
> offset jumps significantly from the previous match.
>

[-- Attachment #2: Type: text/html, Size: 5498 bytes --]

bug#30943: save-hist creates massive cache file

[Eli Zaretskii]

> From: Chris Findeisen <cfindeisen@google.com>
> Date: Thu, 29 Mar 2018 16:27:12 +0000
> Cc: eliz@gnu.org, 30943@debbugs.gnu.org
> 
> Eli, apparently the savehist-trim-history function is only used on
> XEmacs--it's a noop on Emacs.

Granted, I know that, but you mentioned it, so I played along.

bug#30943: save-hist creates massive cache file

[Glenn Morris]

I think it's a reasonable feature request to ask that savehist be
capable of dealing with additional variables that aren't automatically
truncated (or if not it should be explicit about it).

On the other hand, desktop is a superset of savehist and can can already
do this (apparently, see desktop-globals-to-save).

I wonder if load should pay attention to large-file-warning-threshold
somehow?

bug#30943: save-hist creates massive cache file

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: eliz@gnu.org,  30943@debbugs.gnu.org
> Date: Thu, 29 Mar 2018 14:08:30 -0400
> 
> 
> I think it's a reasonable feature request to ask that savehist be
> capable of dealing with additional variables that aren't automatically
> truncated (or if not it should be explicit about it).

That'd require people to register a function to do the truncation.  We
could, of course, provide a couple of default functions to do that
with "simple" variables.

> On the other hand, desktop is a superset of savehist and can can already
> do this (apparently, see desktop-globals-to-save).

Yep.

> I wonder if load should pay attention to large-file-warning-threshold
> somehow?

I actually find that to be a nuisance even for visiting files.  With
64-bit builds, this warning makes very little sense.

bug#30943: save-hist creates massive cache file

[Glenn Morris]

Eli Zaretskii wrote:

> I actually find that to be a nuisance even for visiting files.  With
> 64-bit builds, this warning makes very little sense.

32-bit builds are almost dead, so you may want to revisit the default.
(See trend at https://debbugs.gnu.org/stats/emacs.html )

But isn't it about how much physical memory the system has?
Years ago people were saying 10MB was too small, eg
http://lists.gnu.org/archive/html/emacs-devel/2014-11/msg02358.html

Anyway, getting OT...

The largest el(c) file in the Emacs sources is ja-dic at 2.2MB,
well below the current default for large-file-warning-threshold.
So not warning about loading a 1/2GB file like in the initial report
seems like a disservice.

bug#30943: save-hist creates massive cache file

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: cfindeisen@google.com,  30943@debbugs.gnu.org
> Date: Fri, 30 Mar 2018 01:25:23 -0400
> 
> Eli Zaretskii wrote:
> 
> > I actually find that to be a nuisance even for visiting files.  With
> > 64-bit builds, this warning makes very little sense.
> 
> 32-bit builds are almost dead, so you may want to revisit the default.
> (See trend at https://debbugs.gnu.org/stats/emacs.html )

Yes, I think we should significantly increase the default value of
large-file-warning-threshold.  I always enlarge it in my .emacs,
because the default is even small enough to allow me reading my email
without annoying questions.

> But isn't it about how much physical memory the system has?

That's one factor to consider, yes.  But even that is normally what?
8GB nowadays?  And VM is then 2 or 4 times that?

> The largest el(c) file in the Emacs sources is ja-dic at 2.2MB,
> well below the current default for large-file-warning-threshold.
> So not warning about loading a 1/2GB file like in the initial report
> seems like a disservice.

I wouldn't worry before it got 10 times that, but that's me.

Don't forget that before that file was about to be read, it was
written by the previous Emacs session, which means that previous
session already had all that loaded, and took more memory than that
file's size.  If the user was not worried about their running Emacs's
footprint, why should we annoy them with questions at startup time?

IME, these measures are only useful when they prevent some very grave
problem, like Emacs crashing or becoming completely unresponsive.  (We
had such problems years ago, when the 64-bit build still had bugs with
using integer types that overflowed at INT_MAX, and we indeed had then
tests to prevent users from crossing that border.)  Otherwise, I
personally would prefer _not_ warning about large files and let each
user find the point where it becomes annoying to them.  This is
because IME that point is highly individual, and depends on both user
preferences and the resources on their machine.  We can never do as
well with a fixed threshold, even if its value is somehow calculated
from the available data: there are too many unknown factors here.

Again, just my $0.02.

bug#30943: save-hist creates massive cache file

[Lars Ingebrigtsen]

Glenn Morris <rgm@gnu.org> writes:

> I think it's a reasonable feature request to ask that savehist be
> capable of dealing with additional variables that aren't automatically
> truncated (or if not it should be explicit about it).
>
> On the other hand, desktop is a superset of savehist and can can already
> do this (apparently, see desktop-globals-to-save).

I've now done the same with savehist-additional-variables in Emacs 29.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#30943: save-hist creates massive cache file

[Glenn Morris]

Eli Zaretskii wrote:

> Save-hist doesn't truncate any variables except the ones it knows
> about, because it cannot be sure that truncating some list might leave
> the variable with an invalid value.

savehist doesn't truncate anything AFAICS.
It just assumes variables are trimmed by add-to-history etc in the
course of normal operation. So if you add a non-history variable to
savehist (or set history-length to t), it will eventually blow up.

bug#30943: save-hist creates massive cache file

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Chris Findeisen <cfindeisen@google.com>,  30943@debbugs.gnu.org
> Date: Tue, 27 Mar 2018 15:12:51 -0400
> 
> Eli Zaretskii wrote:
> 
> > Save-hist doesn't truncate any variables except the ones it knows
> > about, because it cannot be sure that truncating some list might leave
> > the variable with an invalid value.
> 
> savehist doesn't truncate anything AFAICS.

I meant savehist-trim-history.  Or maybe I misunderstand what you are
alluding to.

bug#30943: save-hist creates massive cache file

[Glenn Morris]

Eli Zaretskii wrote:

>> savehist doesn't truncate anything AFAICS.
>
> I meant savehist-trim-history.

savehist-trim-history is a no-op on Emacs. savehist relies on the trimming
of history variables that Emacs does in the course of normal operation.