* bug#30943: save-hist creates massive cache file @ 2018-03-25 19:11 Chris Findeisen 2018-03-26 14:59 ` Eli Zaretskii 0 siblings, 1 reply; 14+ messages in thread From: Chris Findeisen @ 2018-03-25 19:11 UTC (permalink / raw) To: 30943 [-- Attachment #1: Type: text/plain, Size: 879 bytes --] save-hist-additional-variables never get truncated by save-hist, leading to a massive cache file and slowdown. Practically, this matters when the cache file silently grows to 1/2 a GB, and emacs begins randomly freezing. history-length is supposed to keep a limit on the max history for save-hist-additional-variables, but it doesn't. In the savehist-save function: (dolist (symbol savehist-minibuffer-history-variables) (when (and (boundp symbol) (not (memq symbol savehist-ignored-variables))) (let ((value (savehist-trim-history (symbol-value symbol))) ;;.... )))) (dolist (symbol savehist-additional-variables) (when (boundp symbol) (let ((value (symbol-value symbol))) (when (savehist-printable value) ;; ... )))) As you can see, the save-hist-trim-history fn is not called in the second code block. [-- Attachment #2: Type: text/html, Size: 16919 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30943: save-hist creates massive cache file 2018-03-25 19:11 bug#30943: save-hist creates massive cache file Chris Findeisen @ 2018-03-26 14:59 ` Eli Zaretskii 2018-03-27 2:50 ` Chris Findeisen 2018-03-27 19:12 ` Glenn Morris 0 siblings, 2 replies; 14+ messages in thread From: Eli Zaretskii @ 2018-03-26 14:59 UTC (permalink / raw) To: Chris Findeisen; +Cc: 30943 > From: Chris Findeisen <cfindeisen@google.com> > Date: Sun, 25 Mar 2018 19:11:21 +0000 > > save-hist-additional-variables never get truncated by save-hist, leading > to a massive cache file and slowdown. Practically, this matters when the cache file silently grows to 1/2 a > GB, and emacs begins randomly freezing. > > history-length is supposed to keep a limit on the max history for save-hist-additional-variables, but it doesn't. > > In the savehist-save function: > > (dolist (symbol savehist-minibuffer-history-variables) > (when (and (boundp symbol) > (not (memq symbol savehist-ignored-variables))) > (let ((value (savehist-trim-history (symbol-value symbol))) > ;;.... > )))) > > (dolist (symbol savehist-additional-variables) > (when (boundp symbol) > (let ((value (symbol-value symbol))) > (when (savehist-printable value) > ;; ... > )))) > > As you can see, the save-hist-trim-history fn is not called in the > second code block. I'm not sure I understand the situation. Is the variable you are trying to save using save-hist a long list or something, and it keeps growing from session to session without any limitation? Save-hist doesn't truncate any variables except the ones it knows about, because it cannot be sure that truncating some list might leave the variable with an invalid value. For variables like minibuffer-history, it knows that truncating the history will produce a history that is still valid, but how can it do the same with other variables? For example, it could be that some other variable needs to be truncated from the other end of the list, otherwise the value will be useless. If I'm missing something about your use case, please tell the details, and perhaps show an example of a variable with which this happens. Thanks. ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30943: save-hist creates massive cache file 2018-03-26 14:59 ` Eli Zaretskii @ 2018-03-27 2:50 ` Chris Findeisen 2018-03-27 19:44 ` Glenn Morris 2018-03-27 19:12 ` Glenn Morris 1 sibling, 1 reply; 14+ messages in thread From: Chris Findeisen @ 2018-03-27 2:50 UTC (permalink / raw) To: eliz; +Cc: 30943 [-- Attachment #1: Type: text/plain, Size: 3864 bytes --] Hey Eli, Correct, the issue is that the variable is growing from session to session without limitation. For instance, some of the culprits from my particular configuration: grep -E -b -o '^\(setq [^ ]+' ~/.emacs.d/.cache/savehist .... 51869:(setq evil-jumps-history 53265:(setq mark-ring 53287:(setq search-ring 53311:(setq regexp-search-ring 53579:(setq extended-command-history I've also seen online this issue crop up for others: https://emacs.stackexchange.com/questions/12086/abnormally-large-savehist-file Save-hist doesn't truncate any variables except the ones it knows > about, because it cannot be sure that truncating some list might leave > the variable with an invalid value. For variables like > minibuffer-history, it knows that truncating the history will produce > a history that is still valid, but how can it do the same with other variables? For example, it could be that some other variable needs to > be truncated from the other end of the list, otherwise the value will > be useless. Hmm, I understand the dilemma about not knowing the format, however there ought to be an alternative way of adding variables that we can "inform" save-hist about. Whether that's demanding/documenting a valid format, "Hey if you add this variable to be tracked by save-hist, it must be formatted like so" or providing a secondary argument that takes a function specifying custom handling, I'm not sure. I think it makes sense to provide some method of honoring the history-length variable. If the list is never culled, any emacs session will just grow over time. That is almost never desired(at least for me), and I would see no point to have additional-variables to be track (I would have to worry about my buffer silently slowing me down). Let me know if I'm missing something, or if you have alternate ideas. Thanks! Regards, Chris Findeisen On Mon, Mar 26, 2018 at 7:59 AM Eli Zaretskii <eliz@gnu.org> wrote: > > From: Chris Findeisen <cfindeisen@google.com> > > Date: Sun, 25 Mar 2018 19:11:21 +0000 > > > > save-hist-additional-variables never get truncated by save-hist, leading > > to a massive cache file and slowdown. Practically, this matters when the > cache file silently grows to 1/2 a > > GB, and emacs begins randomly freezing. > > > > history-length is supposed to keep a limit on the max history for > save-hist-additional-variables, but it doesn't. > > > > In the savehist-save function: > > > > (dolist (symbol savehist-minibuffer-history-variables) > > (when (and (boundp symbol) > > (not (memq symbol savehist-ignored-variables))) > > (let ((value (savehist-trim-history (symbol-value symbol))) > > ;;.... > > )))) > > > > (dolist (symbol savehist-additional-variables) > > (when (boundp symbol) > > (let ((value (symbol-value symbol))) > > (when (savehist-printable value) > > ;; ... > > )))) > > > > As you can see, the save-hist-trim-history fn is not called in the > > second code block. > > I'm not sure I understand the situation. Is the variable you are > trying to save using save-hist a long list or something, and it keeps > growing from session to session without any limitation? > > Save-hist doesn't truncate any variables except the ones it knows > about, because it cannot be sure that truncating some list might leave > the variable with an invalid value. For variables like > minibuffer-history, it knows that truncating the history will produce > a history that is still valid, but how can it do the same with other > variables? For example, it could be that some other variable needs to > be truncated from the other end of the list, otherwise the value will > be useless. > > If I'm missing something about your use case, please tell the details, > and perhaps show an example of a variable with which this happens. > > Thanks. > [-- Attachment #2: Type: text/html, Size: 9723 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30943: save-hist creates massive cache file 2018-03-27 2:50 ` Chris Findeisen @ 2018-03-27 19:44 ` Glenn Morris 2018-03-29 16:27 ` Chris Findeisen 0 siblings, 1 reply; 14+ messages in thread From: Glenn Morris @ 2018-03-27 19:44 UTC (permalink / raw) To: Chris Findeisen; +Cc: 30943 Chris Findeisen wrote: > grep -E -b -o '^\(setq [^ ]+' ~/.emacs.d/.cache/savehist > .... > 51869:(setq evil-jumps-history > 53265:(setq mark-ring > 53287:(setq search-ring > 53311:(setq regexp-search-ring > 53579:(setq extended-command-history None of those look large. You either want to post the whole output, or find the one where the offset jumps significantly from the previous match. ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30943: save-hist creates massive cache file 2018-03-27 19:44 ` Glenn Morris @ 2018-03-29 16:27 ` Chris Findeisen 2018-03-29 17:09 ` Eli Zaretskii 2018-03-29 18:08 ` Glenn Morris 0 siblings, 2 replies; 14+ messages in thread From: Chris Findeisen @ 2018-03-29 16:27 UTC (permalink / raw) To: rgm; +Cc: 30943 [-- Attachment #1: Type: text/plain, Size: 1523 bytes --] > > None of those look large. yeah, I had rm'd my cache file when it was really huge (~1GB), now it's regrowing. To get a sense of what I was looking at previously, I had a similar output to this person's https://emacs.stackexchange.com/questions/12086/abnormally-large-savehist-file. The point is that over time these variables grow and grow and slow. savehist doesn't truncate anything AFAICS. > It just assumes variables are trimmed by add-to-history etc in the > course of normal operation. So if you add a non-history variable to > savehist (or set history-length to t), it will eventually blow up. > Hmm, ok. I may have misunderstood savehist's role then, I thought that savehist managed history size. Eli, apparently the savehist-trim-history function is only used on XEmacs--it's a noop on Emacs. Sorry for the confusion then. First bug report :D ... failed bug report :( . I'll talk to Spacemacs folks about fixing their configuration to guard these variables accordingly. Thanks for your time! Regards, Chris Findeisen On Tue, Mar 27, 2018 at 12:44 PM Glenn Morris <rgm@gnu.org> wrote: > Chris Findeisen wrote: > > > grep -E -b -o '^\(setq [^ ]+' ~/.emacs.d/.cache/savehist > > .... > > 51869:(setq evil-jumps-history > > 53265:(setq mark-ring > > 53287:(setq search-ring > > 53311:(setq regexp-search-ring > > 53579:(setq extended-command-history > > None of those look large. > You either want to post the whole output, or find the one where the > offset jumps significantly from the previous match. > [-- Attachment #2: Type: text/html, Size: 5498 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30943: save-hist creates massive cache file 2018-03-29 16:27 ` Chris Findeisen @ 2018-03-29 17:09 ` Eli Zaretskii 2018-03-29 18:08 ` Glenn Morris 1 sibling, 0 replies; 14+ messages in thread From: Eli Zaretskii @ 2018-03-29 17:09 UTC (permalink / raw) To: Chris Findeisen; +Cc: 30943 > From: Chris Findeisen <cfindeisen@google.com> > Date: Thu, 29 Mar 2018 16:27:12 +0000 > Cc: eliz@gnu.org, 30943@debbugs.gnu.org > > Eli, apparently the savehist-trim-history function is only used on > XEmacs--it's a noop on Emacs. Granted, I know that, but you mentioned it, so I played along. ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30943: save-hist creates massive cache file 2018-03-29 16:27 ` Chris Findeisen 2018-03-29 17:09 ` Eli Zaretskii @ 2018-03-29 18:08 ` Glenn Morris 2018-03-29 18:47 ` Eli Zaretskii 2022-02-15 10:29 ` Lars Ingebrigtsen 1 sibling, 2 replies; 14+ messages in thread From: Glenn Morris @ 2018-03-29 18:08 UTC (permalink / raw) To: Chris Findeisen; +Cc: 30943 I think it's a reasonable feature request to ask that savehist be capable of dealing with additional variables that aren't automatically truncated (or if not it should be explicit about it). On the other hand, desktop is a superset of savehist and can can already do this (apparently, see desktop-globals-to-save). I wonder if load should pay attention to large-file-warning-threshold somehow? ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30943: save-hist creates massive cache file 2018-03-29 18:08 ` Glenn Morris @ 2018-03-29 18:47 ` Eli Zaretskii 2018-03-30 5:25 ` Glenn Morris 2022-02-15 10:29 ` Lars Ingebrigtsen 1 sibling, 1 reply; 14+ messages in thread From: Eli Zaretskii @ 2018-03-29 18:47 UTC (permalink / raw) To: Glenn Morris; +Cc: 30943, cfindeisen > From: Glenn Morris <rgm@gnu.org> > Cc: eliz@gnu.org, 30943@debbugs.gnu.org > Date: Thu, 29 Mar 2018 14:08:30 -0400 > > > I think it's a reasonable feature request to ask that savehist be > capable of dealing with additional variables that aren't automatically > truncated (or if not it should be explicit about it). That'd require people to register a function to do the truncation. We could, of course, provide a couple of default functions to do that with "simple" variables. > On the other hand, desktop is a superset of savehist and can can already > do this (apparently, see desktop-globals-to-save). Yep. > I wonder if load should pay attention to large-file-warning-threshold > somehow? I actually find that to be a nuisance even for visiting files. With 64-bit builds, this warning makes very little sense. ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30943: save-hist creates massive cache file 2018-03-29 18:47 ` Eli Zaretskii @ 2018-03-30 5:25 ` Glenn Morris 2018-03-30 8:14 ` Eli Zaretskii 0 siblings, 1 reply; 14+ messages in thread From: Glenn Morris @ 2018-03-30 5:25 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 30943, cfindeisen Eli Zaretskii wrote: > I actually find that to be a nuisance even for visiting files. With > 64-bit builds, this warning makes very little sense. 32-bit builds are almost dead, so you may want to revisit the default. (See trend at https://debbugs.gnu.org/stats/emacs.html ) But isn't it about how much physical memory the system has? Years ago people were saying 10MB was too small, eg http://lists.gnu.org/archive/html/emacs-devel/2014-11/msg02358.html Anyway, getting OT... The largest el(c) file in the Emacs sources is ja-dic at 2.2MB, well below the current default for large-file-warning-threshold. So not warning about loading a 1/2GB file like in the initial report seems like a disservice. ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30943: save-hist creates massive cache file 2018-03-30 5:25 ` Glenn Morris @ 2018-03-30 8:14 ` Eli Zaretskii 0 siblings, 0 replies; 14+ messages in thread From: Eli Zaretskii @ 2018-03-30 8:14 UTC (permalink / raw) To: Glenn Morris; +Cc: 30943, cfindeisen > From: Glenn Morris <rgm@gnu.org> > Cc: cfindeisen@google.com, 30943@debbugs.gnu.org > Date: Fri, 30 Mar 2018 01:25:23 -0400 > > Eli Zaretskii wrote: > > > I actually find that to be a nuisance even for visiting files. With > > 64-bit builds, this warning makes very little sense. > > 32-bit builds are almost dead, so you may want to revisit the default. > (See trend at https://debbugs.gnu.org/stats/emacs.html ) Yes, I think we should significantly increase the default value of large-file-warning-threshold. I always enlarge it in my .emacs, because the default is even small enough to allow me reading my email without annoying questions. > But isn't it about how much physical memory the system has? That's one factor to consider, yes. But even that is normally what? 8GB nowadays? And VM is then 2 or 4 times that? > The largest el(c) file in the Emacs sources is ja-dic at 2.2MB, > well below the current default for large-file-warning-threshold. > So not warning about loading a 1/2GB file like in the initial report > seems like a disservice. I wouldn't worry before it got 10 times that, but that's me. Don't forget that before that file was about to be read, it was written by the previous Emacs session, which means that previous session already had all that loaded, and took more memory than that file's size. If the user was not worried about their running Emacs's footprint, why should we annoy them with questions at startup time? IME, these measures are only useful when they prevent some very grave problem, like Emacs crashing or becoming completely unresponsive. (We had such problems years ago, when the 64-bit build still had bugs with using integer types that overflowed at INT_MAX, and we indeed had then tests to prevent users from crossing that border.) Otherwise, I personally would prefer _not_ warning about large files and let each user find the point where it becomes annoying to them. This is because IME that point is highly individual, and depends on both user preferences and the resources on their machine. We can never do as well with a fixed threshold, even if its value is somehow calculated from the available data: there are too many unknown factors here. Again, just my $0.02. ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30943: save-hist creates massive cache file 2018-03-29 18:08 ` Glenn Morris 2018-03-29 18:47 ` Eli Zaretskii @ 2022-02-15 10:29 ` Lars Ingebrigtsen 1 sibling, 0 replies; 14+ messages in thread From: Lars Ingebrigtsen @ 2022-02-15 10:29 UTC (permalink / raw) To: Glenn Morris; +Cc: 30943, Chris Findeisen Glenn Morris <rgm@gnu.org> writes: > I think it's a reasonable feature request to ask that savehist be > capable of dealing with additional variables that aren't automatically > truncated (or if not it should be explicit about it). > > On the other hand, desktop is a superset of savehist and can can already > do this (apparently, see desktop-globals-to-save). I've now done the same with savehist-additional-variables in Emacs 29. -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30943: save-hist creates massive cache file 2018-03-26 14:59 ` Eli Zaretskii 2018-03-27 2:50 ` Chris Findeisen @ 2018-03-27 19:12 ` Glenn Morris 2018-03-29 10:49 ` Eli Zaretskii 1 sibling, 1 reply; 14+ messages in thread From: Glenn Morris @ 2018-03-27 19:12 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 30943, Chris Findeisen Eli Zaretskii wrote: > Save-hist doesn't truncate any variables except the ones it knows > about, because it cannot be sure that truncating some list might leave > the variable with an invalid value. savehist doesn't truncate anything AFAICS. It just assumes variables are trimmed by add-to-history etc in the course of normal operation. So if you add a non-history variable to savehist (or set history-length to t), it will eventually blow up. ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30943: save-hist creates massive cache file 2018-03-27 19:12 ` Glenn Morris @ 2018-03-29 10:49 ` Eli Zaretskii 2018-03-29 17:12 ` Glenn Morris 0 siblings, 1 reply; 14+ messages in thread From: Eli Zaretskii @ 2018-03-29 10:49 UTC (permalink / raw) To: Glenn Morris; +Cc: 30943, cfindeisen > From: Glenn Morris <rgm@gnu.org> > Cc: Chris Findeisen <cfindeisen@google.com>, 30943@debbugs.gnu.org > Date: Tue, 27 Mar 2018 15:12:51 -0400 > > Eli Zaretskii wrote: > > > Save-hist doesn't truncate any variables except the ones it knows > > about, because it cannot be sure that truncating some list might leave > > the variable with an invalid value. > > savehist doesn't truncate anything AFAICS. I meant savehist-trim-history. Or maybe I misunderstand what you are alluding to. ^ permalink raw reply [flat|nested] 14+ messages in thread
* bug#30943: save-hist creates massive cache file 2018-03-29 10:49 ` Eli Zaretskii @ 2018-03-29 17:12 ` Glenn Morris 0 siblings, 0 replies; 14+ messages in thread From: Glenn Morris @ 2018-03-29 17:12 UTC (permalink / raw) To: Eli Zaretskii; +Cc: 30943, cfindeisen Eli Zaretskii wrote: >> savehist doesn't truncate anything AFAICS. > > I meant savehist-trim-history. savehist-trim-history is a no-op on Emacs. savehist relies on the trimming of history variables that Emacs does in the course of normal operation. ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2022-02-15 10:29 UTC | newest] Thread overview: 14+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2018-03-25 19:11 bug#30943: save-hist creates massive cache file Chris Findeisen 2018-03-26 14:59 ` Eli Zaretskii 2018-03-27 2:50 ` Chris Findeisen 2018-03-27 19:44 ` Glenn Morris 2018-03-29 16:27 ` Chris Findeisen 2018-03-29 17:09 ` Eli Zaretskii 2018-03-29 18:08 ` Glenn Morris 2018-03-29 18:47 ` Eli Zaretskii 2018-03-30 5:25 ` Glenn Morris 2018-03-30 8:14 ` Eli Zaretskii 2022-02-15 10:29 ` Lars Ingebrigtsen 2018-03-27 19:12 ` Glenn Morris 2018-03-29 10:49 ` Eli Zaretskii 2018-03-29 17:12 ` Glenn Morris
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).