Turning on savehist-mode by default

Turning on savehist-mode by default

[sbaugh]

savehist-mode is a useful mode which is turned on by many Emacs
users. [1] It matches the default behavior of programs like bash and
vim, which save command history by default.  I suggest that we should
find some way to enable savehist by default.

By default savehist creates a timer which saves the history every 5
minutes.  I suggest that by default we should just have behavior like:
(add-hook 'kill-emacs-hook #'savehist-autosave)
If the user explicitly runs (savehist-mode 1) then that would enable the
timer-based saving.

I'm not sure what other issues have prevented savehist from being turned
on by default, but I'm happy to do the work to get it into good shape.

This proposal is somewhat motivated by bug#66993, because project.el
already has some custom code which runs by default which is kind of like
savehist-mode, but which only save project--list.  It may be possible to
get rid of that code in favor of the generic savehist logic, but to do
that we'd need to turn savehist on by default in some form.

[1]: At least, savehist is turned on by every Emacs configuration
framework I've checked, in a brief search of the most popular ones:
https://git.sr.ht/~technomancy/better-defaults/tree/main/item/better-defaults.el
https://github.com/doomemacs/doomemacs/blob/986398504d09e585c7d1a8d73a6394024fe6f164/lisp/doom-editor.el#L356
https://github.com/syl20bnr/spacemacs/blob/a6a834f08af61b5a4dd80e337449f0f7026f6a0c/layers/%2Bspacemacs/spacemacs-defaults/packages.el#L401
https://github.com/bbatsov/prelude/blob/b57ff48e0985a6ef0f1ed9b279ec487c55982334/core/prelude-editor.el#L108
https://github.com/pprevos/emacs-writing-studio/blob/master/init.el#L247
https://systemcrafters.net/emacs-from-scratch/the-best-default-settings/

RE: [External] : Turning on savehist-mode by default

[Drew Adams]

> savehist-mode is a useful mode which is turned on by many Emacs
> users. It matches the default behavior of programs like bash and
> vim, which save command history by default.  I suggest that we
> should find some way to enable savehist by default.

Why?  It's trivial to turn it on.  As you
say, many users (moi aussi) do so.

There are a zillion minor modes that many
users find useful to turn on by default.
It doesn't follow that `emacs -Q' should
turn on any of them by default.

`kill-emacs-hook' should definitely be nil
by default.

> what other issues have prevented savehist
> from being turned on by default

Wrong question.

Re: [External] : Turning on savehist-mode by default

[Philip Kaludercic]

Drew Adams <drew.adams@oracle.com> writes:

>> savehist-mode is a useful mode which is turned on by many Emacs
>> users. It matches the default behavior of programs like bash and
>> vim, which save command history by default.  I suggest that we
>> should find some way to enable savehist by default.
>
> Why?  It's trivial to turn it on.  As you
> say, many users (moi aussi) do so.
>
> There are a zillion minor modes that many
> users find useful to turn on by default.
> It doesn't follow that `emacs -Q' should
> turn on any of them by default.

If something is done by (practically) everyone, especially when it is
something that (practically) all beginners would be interested in, I
wouldn't dismiss the proposal to enable it by default.  How many
features satisfy this condition shouldn't matter.  After all, there are
already a number of features that are enabled by default, like
show-paren-mode, that individually are easy to enable, if you know how
to do it.  The issue is that beginners neither know how to do it, nor
what all the options are that they might be interested in.

IMO the more important question when considering to enable a feature
OOTB, is if it will inconvenience any existing users.  I don't think
there should be any issues with savehist-mode, but I might not know of
some issue?  Been using it myself as well for a while.

On a related topic, I have been running a configuration generator that
has been mentioned here on the list for a while now
(https://emacs.amodernist.com/), that could provide opt-in participation
in statistics of what features people are interested in.  That might
help ground matters like these in somewhat more empirical data, or least
a different perspective.

-- 
Philip Kaludercic

RE: [External] : Turning on savehist-mode by default

[Drew Adams]

> > There are a zillion minor modes that many
> > users find useful to turn on by default.
> > It doesn't follow that `emacs -Q' should
> > turn on any of them by default.
> 
> If something is done by (practically) everyone,

Is it?

> especially when it is something that (practically)
> all beginners would be interested in,

How is that known?

> I wouldn't dismiss the proposal to enable it by default.

I don't dismiss it.

> The issue is that beginners neither know how to do it, nor
> what all the options are that they might be interested in.

And yet it's "done by (practically) everyone"?

___

Let's enable `delete-selection-mode' by default.

It took decades to get `transient-mark-mode' ON
by default.  `delete-selection-mode' completes
that job.  It welcomes new users with the same
type-to-replace behavior they're used to outside
Emacs (everywhere).

Persists nothing.  Is easy for anyone not who
doesn't want it ON to turn OFF.

Re: [External] : Turning on savehist-mode by default

[Philip Kaludercic]

Drew Adams <drew.adams@oracle.com> writes:

>> > There are a zillion minor modes that many
>> > users find useful to turn on by default.
>> > It doesn't follow that `emacs -Q' should
>> > turn on any of them by default.
>> 
>> If something is done by (practically) everyone,
>
> Is it?

Given that my sentence was a hypothetical, I cannot answer this question
since the term "something" cannot be unambiguously instantiated as to be
answered concretely.

>> especially when it is something that (practically)
>> all beginners would be interested in,
>
> How is that known?

It is not known, otherwise there wouldn't really be a discussion.  Our
knowledge can only approximate reality through experience and talking
with different kinds of Emacs users.

Taking the example of savehist-mode, then my experience, which takes
different kinds of users, of different experience levels, people I have
met online and in-person, appears to indicate that this is a popular and
useful feature.

>> I wouldn't dismiss the proposal to enable it by default.
>
> I don't dismiss it.

So you don't oppose enabling savehist-mode by default?

>> The issue is that beginners neither know how to do it, nor
>> what all the options are that they might be interested in.
>
> And yet it's "done by (practically) everyone"?

Let us say, "(practically) everyone" who manages to stay along, by
finding the right options to create a comfortable and productive
environment for themselves.  There are certainly many beginners that
never change this user option; but I suspect that these are also the
ones that never get to taking a look at any user options, because they
give up too soon.

> ___
>
> Let's enable `delete-selection-mode' by default.
>
> It took decades to get `transient-mark-mode' ON
> by default.  `delete-selection-mode' completes
> that job.  It welcomes new users with the same
> type-to-replace behavior they're used to outside
> Emacs (everywhere).
>
> Persists nothing.  Is easy for anyone not who
> doesn't want it ON to turn OFF.

One has to keep in mind that there are a lot of people who use Emacs,
and are familiar with the "feel" of the default key bindings or at least
some subset of these, without having much of an understanding of how to
do things or what is going on.  These are users that should nevertheless
be respected -- hence my point that enabling a feature has to take the
workflow of people into account, for whom a change would break an
expectation.

Note that I am *not* saying that the goal should be to accommodate
newcomers (following whatever current trends may be) at any cost,
especially when this comes at the expense of long-term users.

To make this argument with savehist-mode, one would have to make the
use-case believable, that someone expects the history of mini-buffer
input to not persist between sessions.  I think that is a claim that it
a lot harder to justify, than that inserting a key while a selection is
active, replaces the selection.

-- 
Philip Kaludercic

RE: [External] : Turning on savehist-mode by default

[Drew Adams]

> >> I wouldn't dismiss the proposal to enable it by default.
> >
> > I don't dismiss it.
> 
> So you don't oppose enabling savehist-mode by default?

I don't _dismiss_ it.

___

https://www.wordwebonline.com/search.pl?w=dismiss

WordWeb tells you:

Verb: dismiss

Bar from attention or consideration

"She dismissed his advances";
- disregard, brush aside, brush off [informal], discount, push aside, ignore
 
Cease to consider; put out of judicial consideration

"This case is dismissed!";
- throw out
 
...
 
Declare void

"The President dismissed the parliament and called for new elections";
- dissolve

> >> The issue is that beginners neither know how to do it, nor
> >> what all the options are that they might be interested in.
> >
> > And yet it's "done by (practically) everyone"?
> 
> Let us say, "(practically) everyone" who manages to stay along, by
> finding the right options to create a comfortable and productive
> environment for themselves.  There are certainly many beginners that
> never change this user option; but I suspect that these are also the
> ones that never get to taking a look at any user options, because they
> give up too soon.

So nearly all of those who (1) tried savehist
and (2) didn't give up on getting comfortable
and productive with it by fiddling with its
options.

That's a far cry from "(practically) everyone".
But OK, it's clear now: Most who persisted at
trying to use savehist continue to use it -
that's the claim now.  Might be the case.

> > ___
> >
> > Let's enable `delete-selection-mode' by default.
> >
> > It took decades to get `transient-mark-mode' ON
> > by default.  `delete-selection-mode' completes
> > that job.  It welcomes new users with the same
> > type-to-replace behavior they're used to outside
> > Emacs (everywhere).
> >
> > Persists nothing.  Is easy for anyone not who
> > doesn't want it ON to turn OFF.
> 
> One has to keep in mind that there are a lot of people who use Emacs,
> and are familiar with the "feel" of the default key bindings or at least
> some subset of these, without having much of an understanding of how to
> do things or what is going on.  These are users that should nevertheless
> be respected -- hence my point that enabling a feature has to take the
> workflow of people into account, for whom a change would break an
> expectation.

Not too worry.  One has been keeping this in mind
for many decades...

One kept it in mind about `transient-mark-mode',
yet that was eventually turned ON by default.

(There are still users, today, who turn off
`transient-mark-mode', but likely many fewer than
if the default behavior hadn't changed to ON.)

> Note that I am *not* saying that the goal should be to accommodate
> newcomers (following whatever current trends may be) at any cost,
> especially when this comes at the expense of long-term users.

Nor am I.  Nor was that behind turning ON
`transient-mark-mode' by default.

> To make this argument with savehist-mode, one would have to make the
> use-case believable, that someone expects the history of mini-buffer
> input to not persist between sessions.  I think that is a claim that it
> a lot harder to justify, than that inserting a key while a selection is
> active, replaces the selection.

No, I don't think making that use case believable
is a requirement.  Nor does anyone need to justify
it as a claim.

But FWIW, I do bet that someone - likely even most
users! - currently expect minibuffer input history
to NOT persist.  If it isn't persisted then, hey,
why would someone expect that it is?

To me that expectation _is_ believable.  But I'm
not assuming that "(practically) everyone" turns
on saving of minibuffer histories.  I'm guessing
that more people don't than do.  Just a guess.

__

FWIW, `savehist.el' is a general way to save variable
values, not just a way to save values of minibuffer
history variables.  It's often recommended (including
by me) as a way to persist any variables you like.

Why isn't it written with that foremost in mind, i.e.,
as a general variable-persisting feature?  Why make
users fiddle option `savehist-additional-variables'
to be able to use its (real) functionality?  Why have
pure-Boolean option `savehist-save-minibuffer-history'
instead of just an option whose value is a list of
variables to persist?

Did it start with just the aim of saving minibuffer
histories, and later grafted on the more general
functionality as a wart?  I guess not:

It's not in Emacs 20; I don't have Emacs 21; and in
Emacs 22 there's already the current design, with
variable `savehist-additional-variables'.  The code
comment expressing the motivation says it provides
Emacs with what "many editors (e.g. Vim)" already
had: minibuffer-history persistence.  (Presumably
they had something similar to a minibuffer and its
input histories.)

I understand that it treats minibuffer histories
specially, in that when you use a new history var
that automatically gets included in the list of
vars to persist.  Still, the general functionality
is to persist a set of vars.

__

FWIW, I repeat that I'm a user/fan of savehist.
My value of `savehist-additional-variables':
`(search-ring regexp-search-ring)'.

Re: [External] : Turning on savehist-mode by default

[Eli Zaretskii]

> From: Drew Adams <drew.adams@oracle.com>
> CC: "sbaugh@catern.com" <sbaugh@catern.com>, "emacs-devel@gnu.org"
>  <emacs-devel@gnu.org>
> Date: Sat, 18 Nov 2023 23:01:28 +0000
> 
> > >> I wouldn't dismiss the proposal to enable it by default.
> > >
> > > I don't dismiss it.
> > 
> > So you don't oppose enabling savehist-mode by default?
> 
> I don't _dismiss_ it.

You are splitting hair.  Philip is not a native English speaker, and
he obviously meant "reject" when he said "dismiss".

RE: [External] : Turning on savehist-mode by default

[Drew Adams]

> > > >> I wouldn't dismiss the proposal to enable it by default.
> > > >
> > > > I don't dismiss it.
> > >
> > > So you don't oppose enabling savehist-mode by default?
> >
> > I don't _dismiss_ it.
> 
> You are splitting hair.  Philip is not a native English speaker, and
> he obviously meant "reject" when he said "dismiss".

Did he, obviously?  Then my showing the definition
of `dismiss' might hopefully have helped.

I don't _reject_ it either.  (Not that it's in my
power to do that.  The most I could do is disagree
with it.)

I may in fact be the only person who's even been
considering it.  No one else has spoken up so far.

I haven't seen a good reason to turn it ON by
default - so far.  But maybe there is one.  I've
made clear that I myself use savehist and I often
recommend it.

Persisting minibuffer input has been opt-in, so
far.  I do think it's generally advisable for
users to be aware that what they type will be
persisted, when that's the case.

That applies to input history, search history,
bookmarks, mail drafts, etc.  Automatically
saving predefined thingies (e.g. Customize
settings) is less problematic in this regard
than saving arbitrary typed text/input.

This might be only a minor consideration -
dunno.  What do you think should be considered
when weighing the proposal?

Re: Turning on savehist-mode by default

[Po Lu]

sbaugh@catern.com writes:

> savehist-mode is a useful mode which is turned on by many Emacs
> users. [1] It matches the default behavior of programs like bash and
> vim, which save command history by default.  I suggest that we should
> find some way to enable savehist by default.
>
> By default savehist creates a timer which saves the history every 5
> minutes.  I suggest that by default we should just have behavior like:
> (add-hook 'kill-emacs-hook #'savehist-autosave)
> If the user explicitly runs (savehist-mode 1) then that would enable the
> timer-based saving.
>
> I'm not sure what other issues have prevented savehist from being turned
> on by default, but I'm happy to do the work to get it into good shape.
>
> This proposal is somewhat motivated by bug#66993, because project.el
> already has some custom code which runs by default which is kind of like
> savehist-mode, but which only save project--list.  It may be possible to
> get rid of that code in favor of the generic savehist logic, but to do
> that we'd need to turn savehist on by default in some form.

There is a grave disadvantage to introducing kill-emacs-hooks or saving
files periodically from a timer.  In the case of the former, Emacs will
behave inconsistently, subject to whether it's been killed such that the
hook cannot run.  And the latter runs the risk of files being corrupted
if the file is being saved, by chance, the instant Emacs is killed.

You might object that people are always meant to shut Emacs down in a
way that prompts it to execute these hooks, but I've always considered
it a virtue of Emacs's that it can be so terminated without consequence.

Departing from this in our default configuration would be a step
backwards, inasmuch as we constantly hear of users suffering from long
startup times, implying that Emacs is frequently started for momentary
editing sessions.  And in such scenarios users are apt to kill Emacs by
whatever means are then at hand: xkill, pkill, and the like, this being
behavior that I have myself engaged in on occasion.

As regards enabling savehist-mode by default, I have no strong
objections.  Provided that you can pull it off without either timers or
entries in kill-emacs-hook.

Re: [External] : Turning on savehist-mode by default

[Juri Linkov]

> FWIW, `savehist.el' is a general way to save variable
> values, not just a way to save values of minibuffer
> history variables.  It's often recommended (including
> by me) as a way to persist any variables you like.

Please don't forget there is also `desktop.el'
that is a superset of `savehist.el'

> Why isn't it written with that foremost in mind, i.e.,
> as a general variable-persisting feature?

A general variable-persisting feature already exists:
`define-multisession-variable'.

Re: Turning on savehist-mode by default

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: emacs-devel@gnu.org
> Date: Sun, 19 Nov 2023 14:59:47 +0800
> 
> There is a grave disadvantage to introducing kill-emacs-hooks or saving
> files periodically from a timer.

I disagree that there's a disadvantage in that, let alone a grave one.
I'm using desktop.el, which does both of the above, for many years,
and have yet to see any problems with that.

Re: Turning on savehist-mode by default

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> I disagree that there's a disadvantage in that, let alone a grave one.
> I'm using desktop.el, which does both of the above, for many years,
> and have yet to see any problems with that.

I know this is a disadvantage for one subset of our users, because I
often witness them kill Emacs in precisely the fashion I mentioned
earlier.  And that's if the lock files strewn everywhere belonging to
long-dead Emacs sessions aren't evidence enough.

Re: [External] : Turning on savehist-mode by default

[Yuri Khan]

On Sun, 19 Nov 2023 at 14:11, Juri Linkov <juri@linkov.net> wrote:

> Please don't forget there is also `desktop.el'
> that is a superset of `savehist.el'

Is it though? I’m using ‘desktop-save-mode’, not observing
preservation of histories, and not seeing any user option in the
Desktop group that would shout to me “enable this to save all
histories”. I see ‘file-name-history’ in ‘desktop-globals-to-save’ but
I wouldn’t like to whitelist all the histories I care about.

Re: Turning on savehist-mode by default

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: sbaugh@catern.com,  emacs-devel@gnu.org
> Date: Sun, 19 Nov 2023 15:27:29 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I disagree that there's a disadvantage in that, let alone a grave one.
> > I'm using desktop.el, which does both of the above, for many years,
> > and have yet to see any problems with that.
> 
> I know this is a disadvantage for one subset of our users, because I
> often witness them kill Emacs in precisely the fashion I mentioned
> earlier.  And that's if the lock files strewn everywhere belonging to
> long-dead Emacs sessions aren't evidence enough.

People are free to shoot themselves in the foot, but they shouldn't
then come back to use crying that it hurts.  If someone kills Emacs
with a fatal signal, they should not be surprised that their files are
sometimes corrupted.

Anyway, IME, the files are usable in the next session even if the
previous session was terminated abnormally, like when AC power goes
down.  I don't remember even a single instance of having unusable
files saved by a timer or kill-emacs-hook even though my sessions are
not always terminated normally.

Re: [External] : Turning on savehist-mode by default

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Sun, 19 Nov 2023 14:32:57 +0700
> Cc: Drew Adams <drew.adams@oracle.com>, Philip Kaludercic <philipk@posteo.net>,
>  "sbaugh@catern.com" <sbaugh@catern.com>,
>  "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> 
> On Sun, 19 Nov 2023 at 14:11, Juri Linkov <juri@linkov.net> wrote:
> 
> > Please don't forget there is also `desktop.el'
> > that is a superset of `savehist.el'
> 
> Is it though? I’m using ‘desktop-save-mode’, not observing
> preservation of histories, and not seeing any user option in the
> Desktop group that would shout to me “enable this to save all
> histories”. I see ‘file-name-history’ in ‘desktop-globals-to-save’ but
> I wouldn’t like to whitelist all the histories I care about.

Why not?  I do precisely that, FWIW.  It's a one-time configuration,
so not a lot of effort.

Re: [External] : Turning on savehist-mode by default

[Yuri Khan]

On Sun, 19 Nov 2023 at 15:26, Eli Zaretskii <eliz@gnu.org> wrote:

> > > Please don't forget there is also `desktop.el'
> > > that is a superset of `savehist.el'
> >
> > Is it though? I’m using ‘desktop-save-mode’, not observing
> > preservation of histories, and not seeing any user option in the
> > Desktop group that would shout to me “enable this to save all
> > histories”. I see ‘file-name-history’ in ‘desktop-globals-to-save’ but
> > I wouldn’t like to whitelist all the histories I care about.
>
> Why not?

Why would I? Histories are user data. User data is precious. Losing
user data on a restart is a bug. (Slightly exaggerating.)

I can see three reasons for *not* saving histories: (1) risk of
information disclosure if sensitive data such as passwords is stored
in histories; (2) reduced startup performance if too much history gets
accumulated; and (3) cross-process interference if the application is
typically running in multiple instances. (1) is solved by a blacklist
(opt-out for a few specific histories), (2) by pruning saved histories
to some configurable maximum depth, and (3) by saying Emacs is not
typically used like that[citation needed].

(I remember seeing an implementation of input histories that also
allowed pinning important entries so they would never be pruned, and
explicitly deleting specific entries, right from the UI that displayed
history entries.)

> I do precisely that, FWIW.  It's a one-time configuration,
> so not a lot of effort.

It’s only a one-time configuration if your workflow is constant. If I
start using a feature I had not been using previously, its histories
will initially not be saved, and I’ll have to interrupt my work and
amend my configuration. So it’s a recurring cost.

Re: [External] : Turning on savehist-mode by default

[Eli Zaretskii]

> From: Yuri Khan <yuri.v.khan@gmail.com>
> Date: Sun, 19 Nov 2023 16:06:17 +0700
> Cc: juri@linkov.net, drew.adams@oracle.com, philipk@posteo.net, 
> 	sbaugh@catern.com, emacs-devel@gnu.org
> 
> On Sun, 19 Nov 2023 at 15:26, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > > > Please don't forget there is also `desktop.el'
> > > > that is a superset of `savehist.el'
> > >
> > > Is it though? I’m using ‘desktop-save-mode’, not observing
> > > preservation of histories, and not seeing any user option in the
> > > Desktop group that would shout to me “enable this to save all
> > > histories”. I see ‘file-name-history’ in ‘desktop-globals-to-save’ but
> > > I wouldn’t like to whitelist all the histories I care about.
> >
> > Why not?
> 
> Why would I?

Because some histories you care about, whereas others you don't?
That's the situation here, FWIW.

> Histories are user data. User data is precious. Losing
> user data on a restart is a bug. (Slightly exaggerating.)

The "slightly" part is already exaggerated.

Not all histories are precious enough to save them.  They don't come
for free: you pay with memory for them, if not with longer search
times.

> I can see three reasons for *not* saving histories: (1) risk of
> information disclosure if sensitive data such as passwords is stored
> in histories; (2) reduced startup performance if too much history gets
> accumulated; and (3) cross-process interference if the application is
> typically running in multiple instances. (1) is solved by a blacklist
> (opt-out for a few specific histories), (2) by pruning saved histories
> to some configurable maximum depth, and (3) by saying Emacs is not
> typically used like that[citation needed].

Well, I think you just explained why whitelisting history variables,
and thus using desktop.el for that, is not such an outlandish idea.

One other argument for using desktop.el is if you already use it for
its other features.

> > I do precisely that, FWIW.  It's a one-time configuration,
> > so not a lot of effort.
> 
> It’s only a one-time configuration if your workflow is constant. If I
> start using a feature I had not been using previously, its histories
> will initially not be saved, and I’ll have to interrupt my work and
> amend my configuration. So it’s a recurring cost.

If you start using new features frequently enough, you haven't been
using Emacs long enough ;-)

Let's agree to disagree about the advantages and disadvantages of
whitelisting vs blacklisting.  Basically, both serve the same purpose,
so the decision which one to use depends on the length of the white
list vs the black list.  Which means either method is fine, and
there's no need to represent the other choice as something completely
senseless.

Re: Turning on savehist-mode by default

[Dmitry Gutov]

On 19/11/2023 08:59, Po Lu wrote:
> There is a grave disadvantage to introducing kill-emacs-hooks or saving
> files periodically from a timer.  In the case of the former, Emacs will
> behave inconsistently, subject to whether it's been killed such that the
> hook cannot run.  And the latter runs the risk of files being corrupted
> if the file is being saved, by chance, the instant Emacs is killed.

That would be a real problem if the information saved by savehist-mode 
were of grave importance.

Completion input histories are not usually in that category. People can 
repeat their inputs manually; it's just a time-saving feature.

Re: Turning on savehist-mode by default

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> That would be a real problem if the information saved by savehist-mode
> were of grave importance.
>
> Completion input histories are not usually in that category. People
> can repeat their inputs manually; it's just a time-saving feature.

It is a nuisance for Emacs to display outdated completion history if it
is abruptly killed: anyone who has run multiple bash shells in parallel,
and thus played the guessing game of "which shell's history has been
preserved?" will concur.  Corruption is worse still.

Re: Turning on savehist-mode by default

[Dmitry Gutov]

On 19/11/2023 16:38, Po Lu wrote:
> Dmitry Gutov<dmitry@gutov.dev>  writes:
> 
>> That would be a real problem if the information saved by savehist-mode
>> were of grave importance.
>>
>> Completion input histories are not usually in that category. People
>> can repeat their inputs manually; it's just a time-saving feature.
> It is a nuisance for Emacs to display outdated completion history if it
> is abruptly killed: anyone who has run multiple bash shells in parallel,
> and thus played the guessing game of "which shell's history has been
> preserved?" will concur.  Corruption is worse still.

Yes. But when you play stupid games (e.g. pkill), you win stupid prizes.

Saving the history synchronously has other problems, e.g. exemplified in 
bug#66993.

Re: [External] : Turning on savehist-mode by default

[Spencer Baugh]

Yuri Khan <yuri.v.khan@gmail.com> writes:
> I can see three reasons for *not* saving histories:
>
> (1) risk of information disclosure if sensitive data such as passwords
> is stored in histories;
> (2) reduced startup performance if too much
> history gets accumulated; and
> (3) cross-process interference if the application is typically running
> in multiple instances.
>
> (1) is solved by a blacklist (opt-out for a few specific histories),
> (2) by pruning saved histories to some configurable maximum depth, and
> (3) by saying Emacs is not typically used like that[citation needed].

I agree with your problem statements and solutions.

To add further suggestions: maybe part of a solution for (2) would be
for emacs -Q to not load or save history (e.g. it doesn't enable
savehist-mode).  Possibly even also emacs -q.

That may also help with (3), because 99% of times I have multiple
Emacsen running, it's because I'm running a test emacs -Q or emacs -q.

(3) could also be solved by some intelligent merging of histories; it
would be relatively straightforward to merge the histories when saving
them.  Although probably unnecessary complexity - everyone has been
getting along fine without it.

Re: Turning on savehist-mode by default

[Spencer Baugh]

Po Lu <luangruo@yahoo.com> writes:
> Dmitry Gutov <dmitry@gutov.dev> writes:
>
>> That would be a real problem if the information saved by savehist-mode
>> were of grave importance.
>>
>> Completion input histories are not usually in that category. People
>> can repeat their inputs manually; it's just a time-saving feature.
>
> It is a nuisance for Emacs to display outdated completion history if it
> is abruptly killed:

I agree that outdated completion history is a nuisance.  However, isn't
outdated completion history better than none?

> anyone who has run multiple bash shells in parallel,
> and thus played the guessing game of "which shell's history has been
> preserved?" will concur.

I definitely find the bash history experience extremely annoying.  It
happens even when bash is exited cleanly - it's just a consequence of
the fact that bash overwrites the previous history when exiting, rather
than combining them in some way.

Emacs isn't commonly run in parallel, and when it is, at least for me
there's always one main Emacs instance which outlives the others.  So we
shouldn't have the bash experience.  But still: I'd be happy to
implement some history merging behavior so we don't get the bash history
experience, if that's a requirement for savehist-mode to be turned on by
default.

Re: [External] : Turning on savehist-mode by default

[Visuwesh]

[சனி நவம்பர் 18, 2023] Philip Kaludercic wrote:

>>> especially when it is something that (practically)
>>> all beginners would be interested in,
>>
>> How is that known?
>
> It is not known, otherwise there wouldn't really be a discussion.  Our
> knowledge can only approximate reality through experience and talking
> with different kinds of Emacs users.
>
> Taking the example of savehist-mode, then my experience, which takes
> different kinds of users, of different experience levels, people I have
> met online and in-person, appears to indicate that this is a popular and
> useful feature.
>
> [...]
>>> The issue is that beginners neither know how to do it, nor
>>> what all the options are that they might be interested in.
>>
>> And yet it's "done by (practically) everyone"?
>
> Let us say, "(practically) everyone" who manages to stay along, by
> finding the right options to create a comfortable and productive
> environment for themselves.  There are certainly many beginners that
> never change this user option; but I suspect that these are also the
> ones that never get to taking a look at any user options, because they
> give up too soon.

FWIW, I came to know of savehist-mode fairly late into my Emacs usage
and had no motivation to turn on since most of the time the minibuffer
history is filled with garbage that can be discarded without causing
pain.  If savehist-mode were to be turned on by default, I would
promptly turn it off in my config and call it a day.

I think I would use the mode if it had a whitelist of histories to track
rather than a blacklist, and/or if the history influenced the completion
sorting a la prescient.el.

RE: [External] : Turning on savehist-mode by default

[Drew Adams]

> > FWIW, `savehist.el' is a general way to save variable
> > values, not just a way to save values of minibuffer
> > history variables.  It's often recommended (including
> > by me) as a way to persist any variables you like.
> 
> Please don't forget there is also `desktop.el'
> that is a superset of `savehist.el'

Oh, I don't forget.  That has advantages and
disadvantages compared to savehist.  You can
have any number of desktop files, which can
be in different directories (+).  But it
(always?) saves a lot more than just vars
(-, if you just want to save vars).

> > Why isn't it written with that foremost in mind, i.e.,
> > as a general variable-persisting feature?
> 
> A general variable-persisting feature already exists:
> `define-multisession-variable'.

Looks like that Emacs 29+ feature (like
desktop but in other ways) is also more
than, or different from, just saving vars.

You need to define those vars - and even
access them - in special ways - they're
not ordinary vars.  And it looks like (?)
each such var is stored in its own file
(or in a DB).

I looked at it superficially, however.

It really looks like overkill for just
saving variables.  The use cases we're
talking about, I think, include saving
ordinary variables, of any kind, anytime.

When I mentioned having a simple feature
for just persisting/restoring variables
I meant a simple feature - something like
what savehist does, without separating
out `savehist-additional-variables'.
___

But the existence of multiple ways of
persisting variables suggests that maybe
we should have a doc section about the
topic in the Elisp manual.

That could include simple, ad hoc, saving:
write one or more vars to a file (e.g.
`with-temp-file'); std ways to save a set
of vars (e.g. savehist), std way to define,
save, access `multisession' vars; std ways
to save vars along with other stuff (e.g.
desktop).
___

There are also bookmarks (in Bookmark+, at
least) that just save sets of vars.  You
can have multiple bookmark files, anywhere,
and they can, but need not, be limited to
variable-set bookmarks.  Just jump to a
bookmark to restore a set of vars.

(It's also Bookmark+ that lets you create
desktop files anywhere, including multiple
ones in the same directory.  And lets you
bookmark desktops - just jump to restore.)

RE: [External] : Turning on savehist-mode by default

[Drew Adams]

> > Please don't forget there is also `desktop.el'
> > that is a superset of `savehist.el'
> 
> Is it though? I’m using ‘desktop-save-mode’, not observing
> preservation of histories, and not seeing any user option in the
> Desktop group that would shout to me “enable this to save all
> histories”. I see ‘file-name-history’ in ‘desktop-globals-to-save’ but
> I wouldn’t like to whitelist all the histories I care about.

The two are ... different.  Neither
is a subset of the other.

And even to the extent that part of
one is included in the other, it's
not possible/simple (is it?) to use
the other to get only that subset
behavior.  E.g., you can't (?) use
desktop to only save vars.

Re: Turning on savehist-mode by default

[sbaugh]

Visuwesh <visuweshm@gmail.com> writes:
> I think I would use the mode if it had a whitelist of histories to track
> rather than a blacklist, and/or if the history influenced the completion
> sorting a la prescient.el.

I recently posted a patch which adds an option to sort completion
candidates by the order they appear in the history, in fact, so that
might be available soon (once I rework it for some feedback).  Probably
can't turn that on by default, though.

Re: Turning on savehist-mode by default

[Po Lu]

Spencer Baugh <sbaugh@catern.com> writes:

> I agree that outdated completion history is a nuisance.  However,
> isn't outdated completion history better than none?

Not as I see it.  When M-p is typed and Emacs cannot supply a proper
history entry, it should ding rather than begin iterating over hundreds
of outdated entries.

> I definitely find the bash history experience extremely annoying.  It
> happens even when bash is exited cleanly - it's just a consequence of
> the fact that bash overwrites the previous history when exiting,
> rather than combining them in some way.
>
> Emacs isn't commonly run in parallel, and when it is, at least for me
> there's always one main Emacs instance which outlives the others.  So
> we shouldn't have the bash experience.  But still: I'd be happy to
> implement some history merging behavior so we don't get the bash
> history experience, if that's a requirement for savehist-mode to be
> turned on by default.

Yes.  Despite this not being completely orthodox practice, we know many
users follow it; search for threads on major Emacs forums concerning
startup time, and you will recognize the pervasiveness of short,
ephemeral Emacs sessions.

Re: Turning on savehist-mode by default

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> Yes. But when you play stupid games (e.g. pkill), you win stupid
> prizes.

There are't such "prizes" in the default installation now, so we should
strive not to introduce any.

Re: Turning on savehist-mode by default

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > savehist-mode is a useful mode which is turned on by many Emacs
  > users. [1] It matches the default behavior of programs like bash and
  > vim, which save command history by default.  I suggest that we should
  > find some way to enable savehist by default.

I suppose this functionality would not bother anyone, but would it
cause annoying pauses sometimes?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Turning on savehist-mode by default

[sbaugh]

Po Lu <luangruo@yahoo.com> writes:
> Spencer Baugh <sbaugh@catern.com> writes:
>
>> I agree that outdated completion history is a nuisance.  However,
>> isn't outdated completion history better than none?
>
> Not as I see it.  When M-p is typed and Emacs cannot supply a proper
> history entry, it should ding rather than begin iterating over hundreds
> of outdated entries.

So do you yourself not use savehist-mode?  For these reasons?  Can you
describe your usage pattern that makes savehist-mode not work for you?

Re: Turning on savehist-mode by default

[Spencer Baugh]

Richard Stallman <rms@gnu.org> writes:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > savehist-mode is a useful mode which is turned on by many Emacs
>   > users. [1] It matches the default behavior of programs like bash and
>   > vim, which save command history by default.  I suggest that we should
>   > find some way to enable savehist by default.
>
> I suppose this functionality would not bother anyone, but would it
> cause annoying pauses sometimes?

Probably not, but to be extra sure, I suggest only turning on the
kill-emacs-hook part of savehist-mode by default, not the timer.

Re: Turning on savehist-mode by default

[Po Lu]

sbaugh@catern.com writes:

> So do you yourself not use savehist-mode?  For these reasons?  Can you
> describe your usage pattern that makes savehist-mode not work for you?

I thought I already related why: it doesn't accommodate running multiple
Emacs sessions at once or Emacs processes being abruptly killed.  If it
is to be enabled by default, both these deficiencies must be addressed.

Re: [OT] Not clobbering bash history

[brickviking]

[-- Attachment #1: Type: text/plain, Size: 1900 bytes --]

On Mon, 20 Nov 2023 at 04:18, Spencer Baugh <sbaugh@catern.com> wrote:

> Po Lu <luangruo@yahoo.com> writes:
> > anyone who has run multiple bash shells in parallel,
> > and thus played the guessing game of "which shell's history has been
> > preserved?" will concur.
>
> I definitely find the bash history experience extremely annoying.  It
> happens even when bash is exited cleanly - it's just a consequence of
> the fact that bash overwrites the previous history when exiting, rather
> than combining them in some way.
>
>
Just as a complete aside for preventing bash history erasure, I got around
it like this:

* shopt -s histappend # append to the history file, don't overwrite it
* HISTIGNORE='cd "`printf "%b"*' # Need this so tab filename completion
lines don't get stored
* export HISTCONTROL=ignoreboth # I could also have used erasedups here but
felt this would be unacceptably slow
* export HISTSIZE=30000000 # Yes, it's huge, but I get history going back
years

I prevent history erasure by the simple process of defining history to be
huge (30Mb) and save each line as I hit enter without trimming the
.bash_history shorter. The one disadvantage I've struck so far is that I
have very long bash startup times. I get around that to a degree by
periodically trimming history down to a more sane 10,000 lines after
archiving to a master file. I also load 1k lines of history each time I
start up a shell, hence the longer startup times.

Relating to history searches, I tend to use grep "my pattern"
~/.bash_history first, and if I don't find what I want there (a really
unusual occurrence now) I grep the master file as well. I have history
going back several years, and that has saved my time more times than I can
remember.

I hope that's a useful diversion from the intricacies of emacs history
saving, and might give a tip or two to those bash users among us.

Regards, brickviking

[-- Attachment #2: Type: text/html, Size: 2397 bytes --]

Re: [External] : Turning on savehist-mode by default

[Manuel Giraud via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Yuri Khan <yuri.v.khan@gmail.com>
>> Date: Sun, 19 Nov 2023 14:32:57 +0700
>> Cc: Drew Adams <drew.adams@oracle.com>, Philip Kaludercic <philipk@posteo.net>,
>>  "sbaugh@catern.com" <sbaugh@catern.com>,
>>  "emacs-devel@gnu.org" <emacs-devel@gnu.org>
>> 
>> On Sun, 19 Nov 2023 at 14:11, Juri Linkov <juri@linkov.net> wrote:
>> 
>> > Please don't forget there is also `desktop.el'
>> > that is a superset of `savehist.el'
>> 
>> Is it though? I’m using ‘desktop-save-mode’, not observing
>> preservation of histories, and not seeing any user option in the
>> Desktop group that would shout to me “enable this to save all
>> histories”. I see ‘file-name-history’ in ‘desktop-globals-to-save’ but
>> I wouldn’t like to whitelist all the histories I care about.
>
> Why not?  I do precisely that, FWIW.  It's a one-time configuration,
> so not a lot of effort.

I'm curious.  What desktop save option did you set for this?  I am
currently using desktop-save *and* savehist but I could use just
desktop-save?
-- 
Manuel Giraud

Re: [External] : Turning on savehist-mode by default

[Eli Zaretskii]

> From: Manuel Giraud <manuel@ledu-giraud.fr>
> Cc: Yuri Khan <yuri.v.khan@gmail.com>,  juri@linkov.net,
>   drew.adams@oracle.com,  philipk@posteo.net,  sbaugh@catern.com,
>   emacs-devel@gnu.org
> Date: Mon, 20 Nov 2023 10:53:46 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Is it though? I’m using ‘desktop-save-mode’, not observing
> >> preservation of histories, and not seeing any user option in the
> >> Desktop group that would shout to me “enable this to save all
> >> histories”. I see ‘file-name-history’ in ‘desktop-globals-to-save’ but
> >> I wouldn’t like to whitelist all the histories I care about.
> >
> > Why not?  I do precisely that, FWIW.  It's a one-time configuration,
> > so not a lot of effort.
> 
> I'm curious.  What desktop save option did you set for this?

I customize desktop-globals-to-save.  It has file-name-history by
default, and I add other history variables to it.

> I am currently using desktop-save *and* savehist but I could use
> just desktop-save?

I didn't study savehist too deeply, but it sounds like these
functionalities are equivalent, yes.

Re: [External] : Turning on savehist-mode by default

[Manuel Giraud via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Manuel Giraud <manuel@ledu-giraud.fr>
>> Cc: Yuri Khan <yuri.v.khan@gmail.com>,  juri@linkov.net,
>>   drew.adams@oracle.com,  philipk@posteo.net,  sbaugh@catern.com,
>>   emacs-devel@gnu.org
>> Date: Mon, 20 Nov 2023 10:53:46 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> Is it though? I’m using ‘desktop-save-mode’, not observing
>> >> preservation of histories, and not seeing any user option in the
>> >> Desktop group that would shout to me “enable this to save all
>> >> histories”. I see ‘file-name-history’ in ‘desktop-globals-to-save’ but
>> >> I wouldn’t like to whitelist all the histories I care about.
>> >
>> > Why not?  I do precisely that, FWIW.  It's a one-time configuration,
>> > so not a lot of effort.
>> 
>> I'm curious.  What desktop save option did you set for this?
>
> I customize desktop-globals-to-save.  It has file-name-history by
> default, and I add other history variables to it.

Thanks, I'll try to do that.  BTW, maybe this requires a paragraph in
info.  The "(emacs) Saving Emacs Sessions" node, only refers to the
savehist library.

>> I am currently using desktop-save *and* savehist but I could use
>> just desktop-save?
>
> I didn't study savehist too deeply, but it sounds like these
> functionalities are equivalent, yes.

Looking at my ".emacs.d/history" file, it seems that savehist comes with
lots of history saved by default: erc, eww, gnus-search, query-replace…
just to name a few.
-- 
Manuel Giraud

Re: [External] : Turning on savehist-mode by default

[Eli Zaretskii]

> From: Manuel Giraud <manuel@ledu-giraud.fr>
> Cc: yuri.v.khan@gmail.com,  juri@linkov.net,  drew.adams@oracle.com,
>   philipk@posteo.net,  sbaugh@catern.com,  emacs-devel@gnu.org
> Date: Mon, 20 Nov 2023 14:15:24 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I customize desktop-globals-to-save.  It has file-name-history by
> > default, and I add other history variables to it.
> 
> Thanks, I'll try to do that.  BTW, maybe this requires a paragraph in
> info.  The "(emacs) Saving Emacs Sessions" node, only refers to the
> savehist library.

Maybe.  Let me think about it.

> >> I am currently using desktop-save *and* savehist but I could use
> >> just desktop-save?
> >
> > I didn't study savehist too deeply, but it sounds like these
> > functionalities are equivalent, yes.
> 
> Looking at my ".emacs.d/history" file, it seems that savehist comes with
> lots of history saved by default: erc, eww, gnus-search, query-replace…
> just to name a few.

If you really want all or most of the history variables, then savehist
is easier and more convenient, since you don't have to name every
variable you want to save.  Me, I want to save only a small number of
them, the rest are useless for me.

Re: Turning on savehist-mode by default

[Spencer Baugh]

Po Lu <luangruo@yahoo.com> writes:
> sbaugh@catern.com writes:
>
>> So do you yourself not use savehist-mode?  For these reasons?  Can you
>> describe your usage pattern that makes savehist-mode not work for you?
>
> I thought I already related why: it doesn't accommodate running multiple
> Emacs sessions at once or Emacs processes being abruptly killed.  If it
> is to be enabled by default, both these deficiencies must be addressed.

Yes, I understand this point.

What I mean is: Can you describe, concretely, when you personally are
running multiple Emacs sessions at once, or abruptly killing an Emacs
process?

I ask just to make sure I have at least one concrete example of the kind
of usage should be supported.  Otherwise we might come up with a
solution that doesn't actually work for you.

Plus, since such usage seems to be considered uncommon, a concrete
example will further support your point.

RE: [External] : Turning on savehist-mode by default

[Drew Adams]

> I customize desktop-globals-to-save.  It has file-name-history by
> default, and I add other history variables to it.

FWIW, that option shouldn't be named `...globals...'.
It should be named something like `...variables...'
or `...vars...'.

Re: Not clobbering bash history

[Juri Linkov]

> Just as a complete aside for preventing bash history erasure, I got around
> it like this:
>
> * export HISTCONTROL=ignoreboth # I could also have used erasedups here but
> felt this would be unacceptably slow

Actually this is not quite off-topic.  In Emacs, the corresponding option
is 'history-delete-duplicates' that corresponds to 'ignoredups'.
But unfortunately still no option corresponding to 'ignorespace'.

Instead of using such advice

  (define-advice add-to-history (:around (ofun history-var newelt &rest args) ignorespace)
    (unless (string-match-p "\\` " newelt)
      (apply ofun history-var newelt args)))

maybe 'history-delete-duplicates' should support a new value 'ignorespace'?

> * export HISTSIZE=30000000
> # Yes, it's huge, but I get history going back years

I found that it helps to reduce the size of the shell history
by splitting the history file by different file trees using advice:

  (define-advice shell (:around (ofun &rest args) per-tree-history)
    (let ((process-environment (copy-sequence process-environment))
          (root (locate-dominating-file default-directory ".bash_history")))
      (when root
        (setenv "HISTFILE" (expand-file-name ".bash_history" root)))
      (apply ofun args)))

but no idea how this could be more customizable.

Re: [External] : Turning on savehist-mode by default

[Eli Zaretskii]

> From: Drew Adams <drew.adams@oracle.com>
> CC: "yuri.v.khan@gmail.com" <yuri.v.khan@gmail.com>,
>         "juri@linkov.net"
> 	<juri@linkov.net>,
>         "philipk@posteo.net" <philipk@posteo.net>,
>         "sbaugh@catern.com" <sbaugh@catern.com>,
>         "emacs-devel@gnu.org"
> 	<emacs-devel@gnu.org>
> Date: Mon, 20 Nov 2023 16:57:16 +0000
> 
> > I customize desktop-globals-to-save.  It has file-name-history by
> > default, and I add other history variables to it.
> 
> FWIW, that option shouldn't be named `...globals...'.
> It should be named something like `...variables...'
> or `...vars...'.

There's also desktop-locals-to-save.

RE: [External] : Turning on savehist-mode by default

[Drew Adams]

> > > ... desktop-globals-to-save ...
> >
> > FWIW, that option shouldn't be named `...globals...'.
> > It should be named something like `...variables...'
> > or `...vars...'.
> 
> There's also desktop-locals-to-save.

"globals" and "locals" isn't clear, IMHO, and
isn't very discoverable (e.g. with apropos).

They should be named something like
`...global-vars...' and `...buffer-local-vars...'.

"Local" variables as opposed to global doesn't
necessarily (or even normally) mean _buffer_
local.  (But saving other kinds of local vars
doesn't make sense, admittedly.)

More generally, anything having to do with
"buffer local" should include "buffer" in the
name, especially if the context doesn't make
clear that this is what's meant.

Same for directory-local, file-local, and
connection-local vars.

Note that at least the options etc. for those
use the word "variables" in the name.  We don't
use a name such as just `ignored-locals'.

It is, however, true that we've omitted `file'
from the names of several file-local names,
such as `ignored-local-variables'.  It would
be better for those too to include `file' in
the name. 

Re: [External] : Turning on savehist-mode by default

[Eli Zaretskii]

> From: Drew Adams <drew.adams@oracle.com>
> CC: "manuel@ledu-giraud.fr" <manuel@ledu-giraud.fr>,
>         "yuri.v.khan@gmail.com"
> 	<yuri.v.khan@gmail.com>,
>         "juri@linkov.net" <juri@linkov.net>,
>         "philipk@posteo.net" <philipk@posteo.net>,
>         "sbaugh@catern.com"
> 	<sbaugh@catern.com>,
>         "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> Date: Mon, 20 Nov 2023 18:54:47 +0000
> 
> > > > ... desktop-globals-to-save ...
> > >
> > > FWIW, that option shouldn't be named `...globals...'.
> > > It should be named something like `...variables...'
> > > or `...vars...'.
> > 
> > There's also desktop-locals-to-save.
> 
> "globals" and "locals" isn't clear, IMHO,

They are crystal clear IMO.

> and isn't very discoverable (e.g. with apropos).

I don't see why they should be discoverable via apropos.  They are
minor features of desktop.el, and therefore should be discovered
either via customize-group or by simply searching desktop.el for
"defcustom".

Re: [OT] Not clobbering bash history

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > I definitely find the bash history experience extremely annoying.  It
  > > happens even when bash is exited cleanly - it's just a consequence of
  > > the fact that bash overwrites the previous history when exiting, rather
  > > than combining them in some way.

Should we suggest that the Bash developer add a feature to handle this
case (multiple shells in parallel) the "right" way?  If many users
would like it, that could make it worth building in.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [OT] Not clobbering bash history

[Arsen Arsenović]

[-- Attachment #1: Type: text/plain, Size: 911 bytes --]


Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > I definitely find the bash history experience extremely annoying.  It
>   > > happens even when bash is exited cleanly - it's just a consequence of
>   > > the fact that bash overwrites the previous history when exiting, rather
>   > > than combining them in some way.
>
> Should we suggest that the Bash developer add a feature to handle this
> case (multiple shells in parallel) the "right" way?  If many users
> would like it, that could make it worth building in.

That would be very nice.  This issue has almost been prolific enough to
force me to switch shells.

Thanks, have a lovely night.
-- 
Arsen Arsenović

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

Re: [OT] Not clobbering bash history

[Jens Schmidt via Emacs news and miscellaneous discussions outside the scope of other Emacs mailing lists]

Moving to tangents ... please CC me.

On 2023-11-22  04:32, Arsen Arsenović wrote:

>> Should we suggest that the Bash developer add a feature to handle this
>> case (multiple shells in parallel) the "right" way?  If many users
>> would like it, that could make it worth building in.
> 
> That would be very nice.  This issue has almost been prolific enough to
> force me to switch shells.

This Bash-bashing made me curious, as I do not remember having issues with
loss of its history, ever.  And that without such tricks as using an ever
growing history file.  I just use "shopt -s histappend" and 4096 lines as
HISTFILESIZE.  Plus I close my Bashes orderly before shutting down.  Plus
I do not use Bash from Emacs, only in "real" terminals.  Eh, plus I rarely
use nested Bashes.

So given all that, is there a reproducer for this?

---
via emacs-tangents mailing list (https://lists.gnu.org/mailman/listinfo/emacs-tangents)

Re: [OT] Not clobbering bash history

[Arsen Arsenović via Emacs news and miscellaneous discussions outside the scope of other Emacs mailing lists]

[-- Attachment #1.1: Type: text/plain, Size: 1713 bytes --]


Jens Schmidt <jschmidt4gnu@vodafonemail.de> writes:

> Moving to tangents ... please CC me.
>
> On 2023-11-22  04:32, Arsen Arsenović wrote:
>
>>> Should we suggest that the Bash developer add a feature to handle this
>>> case (multiple shells in parallel) the "right" way?  If many users
>>> would like it, that could make it worth building in.
>> 
>> That would be very nice.  This issue has almost been prolific enough to
>> force me to switch shells.
>
> This Bash-bashing made me curious, as I do not remember having issues
> with loss of its history, ever.

I do not partake in bash-bashing enthusiastically.  I'm a big fan of
readline and appreciate how reliable bash is.

> And that without such tricks as using an ever growing history file.  I
> just use "shopt -s histappend" and 4096 lines as HISTFILESIZE.  Plus I
> close my Bashes orderly before shutting down.  Plus I do not use Bash
> from Emacs, only in "real" terminals.  Eh, plus I rarely use nested
> Bashes.
>
> So given all that, is there a reproducer for this?

Unfortunately, I have been unable to produce a decent set of
reproduction steps.  This issue happens (relatively) infrequently and
inexplicably on my machines.  It pains me to complain without solid
information.

I also have shopt -s histappend set.

I suspect that something starts a bash process and does not set
histappend (--norc?), leading bash to override history.

I wonder whether, in general, if bash (libhistory?) was written with the
assumption of multiple processes writing to the history file, so that it
keeps its history in sync with other bash processes, problems of this
category would go away.
-- 
Arsen Arsenović

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 381 bytes --]

[-- Attachment #2: Type: text/plain, Size: 92 bytes --]

---
via emacs-tangents mailing list (https://lists.gnu.org/mailman/listinfo/emacs-tangents)

Re: [OT] Not clobbering bash history

[Spencer Baugh]

Arsen Arsenović <arsen@aarsen.me> writes:
> Jens Schmidt <jschmidt4gnu@vodafonemail.de> writes:
>
>> Moving to tangents ... please CC me.
>>
>> On 2023-11-22  04:32, Arsen Arsenović wrote:
>>
>>>> Should we suggest that the Bash developer add a feature to handle this
>>>> case (multiple shells in parallel) the "right" way?  If many users
>>>> would like it, that could make it worth building in.
>>> 
>>> That would be very nice.  This issue has almost been prolific enough to
>>> force me to switch shells.
>>
>> This Bash-bashing made me curious, as I do not remember having issues
>> with loss of its history, ever.
>
> I do not partake in bash-bashing enthusiastically.  I'm a big fan of
> readline and appreciate how reliable bash is.
>
>> And that without such tricks as using an ever growing history file.  I
>> just use "shopt -s histappend" and 4096 lines as HISTFILESIZE.  Plus I
>> close my Bashes orderly before shutting down.  Plus I do not use Bash
>> from Emacs, only in "real" terminals.  Eh, plus I rarely use nested
>> Bashes.
>>
>> So given all that, is there a reproducer for this?
>
> Unfortunately, I have been unable to produce a decent set of
> reproduction steps.  This issue happens (relatively) infrequently and
> inexplicably on my machines.  It pains me to complain without solid
> information.
>
> I also have shopt -s histappend set.
>
> I suspect that something starts a bash process and does not set
> histappend (--norc?), leading bash to override history.

I've always suspected TRAMP of this.

---
via emacs-tangents mailing list (https://lists.gnu.org/mailman/listinfo/emacs-tangents)

Re: [External] : Turning on savehist-mode by default

[Eli Zaretskii]

> Date: Mon, 20 Nov 2023 16:05:09 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: yuri.v.khan@gmail.com, juri@linkov.net, drew.adams@oracle.com,
>  philipk@posteo.net, sbaugh@catern.com, emacs-devel@gnu.org
> 
> > From: Manuel Giraud <manuel@ledu-giraud.fr>
> > Cc: yuri.v.khan@gmail.com,  juri@linkov.net,  drew.adams@oracle.com,
> >   philipk@posteo.net,  sbaugh@catern.com,  emacs-devel@gnu.org
> > Date: Mon, 20 Nov 2023 14:15:24 +0100
> > 
> > Eli Zaretskii <eliz@gnu.org> writes:
> > 
> > > I customize desktop-globals-to-save.  It has file-name-history by
> > > default, and I add other history variables to it.
> > 
> > Thanks, I'll try to do that.  BTW, maybe this requires a paragraph in
> > info.  The "(emacs) Saving Emacs Sessions" node, only refers to the
> > savehist library.
> 
> Maybe.  Let me think about it.

Now done on the emacs-29 branch.

Re: [External] : Turning on savehist-mode by default

[Manuel Giraud via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

[...]

>> > Thanks, I'll try to do that.  BTW, maybe this requires a paragraph in
>> > info.  The "(emacs) Saving Emacs Sessions" node, only refers to the
>> > savehist library.
>> 
>> Maybe.  Let me think about it.
>
> Now done on the emacs-29 branch.

Thanks!
-- 
Manuel Giraud

Re: [OT] Not clobbering bash history

[Jens Schmidt via Emacs news and miscellaneous discussions outside the scope of other Emacs mailing lists]

On 2023-11-23  13:42, Spencer Baugh wrote:
> Arsen Arsenović <arsen@aarsen.me> writes:
>
>> Unfortunately, I have been unable to produce a decent set of
>> reproduction steps.  This issue happens (relatively) infrequently and
>> inexplicably on my machines.  It pains me to complain without solid
>> information.
>>
>> I also have shopt -s histappend set.
>>
>> I suspect that something starts a bash process and does not set
>> histappend (--norc?), leading bash to override history.
> 
> I've always suspected TRAMP of this.

Arsen's problem description rang a bell ... on my development server
at work I configure a non-standard history file in my .bashrc.  And
regardless of that, every now and then, the default ~/.bash_history
gets created.

So probably a work-around for your overwrite problems could be exactly
that: Just configure HISTFILE=~/.my_hidden_bash_history near the same
spot where you set 'shopt -s histappend'.  Any villain not respecting
the latter won't (hopefully) respect the former as well, leaving your
non-default Bash history untouched.  (Unless you have tried that
already, of course.)

Besides that, the contents and timestamp of the default ~/.bash_history,
should it still get created, could then help finding the villain.


---
via emacs-tangents mailing list (https://lists.gnu.org/mailman/listinfo/emacs-tangents)

Re: [OT] Not clobbering bash history

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Should we suggest that the Bash developer add a feature to handle this
  > > case (multiple shells in parallel) the "right" way?  If many users
  > > would like it, that could make it worth building in.

  > That would be very nice.  This issue has almost been prolific enough to
  > force me to switch shells.

How about if you (or someone) write a description of the behavior you
would like Bash to offer (at least as an option)?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [OT] Not clobbering bash history

[Arsen Arsenović]

[-- Attachment #1: Type: text/plain, Size: 1679 bytes --]


Richard Stallman <rms@gnu.org> writes:

>   > > Should we suggest that the Bash developer add a feature to handle this
>   > > case (multiple shells in parallel) the "right" way?  If many users
>   > > would like it, that could make it worth building in.
>
>   > That would be very nice.  This issue has almost been prolific enough to
>   > force me to switch shells.
>
> How about if you (or someone) write a description of the behavior you
> would like Bash to offer (at least as an option)?

Sure.

libhistory should, on recall attempt, try to re-read the history file
from the last point it was on in order to catch new histories, it should
append to the file, and it should attempt to lock the file via flock or
similar if such facilities are available (just in case).

Bash already implements this partially, lacking the update-on-recall
functionality (I believe ZSH calls its version of this "share_history"),
and lacking a way to append a history command *before* it is executed,
to my awareness.

In the event of truncation, libhistory needs to be careful not to lose
any histories that were to be submitted in between the moment of
determination of truncation and commitment of the truncation to disk.

On occasion, I wish histories were stored in a SQLite3 database :-)

As a QoL feature, bash should prevent history truncation if ran with
--norc or other flags that would inhibit HISTFILE being set potentially.
I suspect that this latter change alone would prevent the issue I was
seeing (but I can't be sure, since I never caught bash in the act of
history file truncation).

Thanks, have a lovely day.
-- 
Arsen Arsenović

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 381 bytes --]

Re: Turning on savehist-mode by default

[Thanos Apollo]

[-- Attachment #1: Type: text/plain, Size: 1294 bytes --]


sbaugh@catern.com writes:

> savehist-mode is a useful mode which is turned on by many Emacs
> users. [1] It matches the default behavior of programs like bash and
> vim, which save command history by default.  I suggest that we should
> find some way to enable savehist by default.
>

What is the recommended way for package authors to disable
savehist-mode?

If savehist-mode is enabled by default, it is advisable to also consider
a simple way to disable it for a specific package mode.

for example I'm using emacs to answer a MCQ, such as:

Q:Which one of the following statements concerning the
ability of acidosis to precipitate a crisis in sickle cell
anemia is correct?

```emacs-lisp
(completing-read "Answer: "
'("Fetal blood has a higher affinity for oxygen than does
adult blood because Hb F has a decreased affinity for
2,3-BPG."
"Purified Hb F (stripped of 2,3-BPG) has a higher affin-
ity for oxygen than does purified Hb A."
"The globin chain composition of Hb F is α2δ2."))
```
Saving the latest and usually correct answer defeats the purpose of an MCQ.

One workaround for this is to set minibuffer-history to nil, although it
is unclear whether this is advised, even with a let statement.

-- 
Thanos Apollo
https://thanosapollo.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 865 bytes --]

Re: Turning on savehist-mode by default

[Thanos Apollo]

[-- Attachment #1: Type: text/plain, Size: 587 bytes --]


Thanos Apollo <public@thanosapollo.org> writes:

> [[PGP Signed Part:Undecided]]
>
> sbaugh@catern.com writes:
>
>> savehist-mode is a useful mode which is turned on by many Emacs
>> users. [1] It matches the default behavior of programs like bash and
>> vim, which save command history by default.  I suggest that we should
>> find some way to enable savehist by default.
>>
>
> What is the recommended way for package authors to disable
> savehist-mode?

Ignore this email, my question had nothing to do with savehist.

-- 
Thanos Apollo
https://thanosapollo.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 865 bytes --]

Re: Turning on savehist-mode by default

[Eli Zaretskii]

> From: Thanos Apollo <public@thanosapollo.org>
> Date: Tue, 28 Nov 2023 13:04:41 +0200
> 
> > savehist-mode is a useful mode which is turned on by many Emacs
> > users. [1] It matches the default behavior of programs like bash and
> > vim, which save command history by default.  I suggest that we should
> > find some way to enable savehist by default.
> >
> 
> What is the recommended way for package authors to disable
> savehist-mode?
> 
> If savehist-mode is enabled by default, it is advisable to also consider
> a simple way to disable it for a specific package mode.
> 
> for example I'm using emacs to answer a MCQ, such as:
> 
> Q:Which one of the following statements concerning the
> ability of acidosis to precipitate a crisis in sickle cell
> anemia is correct?
> 
> ```emacs-lisp
> (completing-read "Answer: "
> '("Fetal blood has a higher affinity for oxygen than does
> adult blood because Hb F has a decreased affinity for
> 2,3-BPG."
> "Purified Hb F (stripped of 2,3-BPG) has a higher affin-
> ity for oxygen than does purified Hb A."
> "The globin chain composition of Hb F is α2δ2."))
> ```
> Saving the latest and usually correct answer defeats the purpose of an MCQ.
> 
> One workaround for this is to set minibuffer-history to nil, although it
> is unclear whether this is advised, even with a let statement.

You could instead bind history-add-new-input to nil around calls to
completing-read.

Re: Turning on savehist-mode by default

[Adam Porter]

FWIW, IME, it may not be a good idea to enable savehist-mode by default. 
  As an example, see this issue report on Ement.el's repository:

https://github.com/alphapapa/ement.el/issues/216

The gist is that, for users who enabled savehist, it was saving the 
variable command-history, which included some complex commands whose 
arguments (from interactive completion) included large data structures 
(which are hidden from the user when called interactively).  This 
resulted in savehist files hundreds of MB in size and very noticeable 
pauses whenever savehist saved its data, from timers and upon Emacs exit.

The worst part was how long it took to discover the source of the 
problem.  There were intermittent reports of weird pauses for months, 
without any discernible reason, until a user finally narrowed it down to 
using savehist.  Then we had to add this hacky workaround in Ement to 
try to automatically avoid the problem.

Such cases may be rare (or not--I can't say), but when it happens, it 
seems to be a bear to deal with, being very unintuitive.  For myself, 
even having written a number of Emacs packages, I wasn't yet aware of 
the command-history variable and how it saves arguments, and certainly 
not the implications of using savehist as well.

--Adam

Re: Turning on savehist-mode by default

[Dmitry Gutov]

On 28/11/2023 23:07, Adam Porter wrote:
> FWIW, IME, it may not be a good idea to enable savehist-mode by default. 
>   As an example, see this issue report on Ement.el's repository:
> 
> https://github.com/alphapapa/ement.el/issues/216
> 
> The gist is that, for users who enabled savehist, it was saving the 
> variable command-history, which included some complex commands whose 
> arguments (from interactive completion) included large data structures 
> (which are hidden from the user when called interactively).  This 
> resulted in savehist files hundreds of MB in size and very noticeable 
> pauses whenever savehist saved its data, from timers and upon Emacs exit.

savehist-mode is very popular, so even if it's not on by default, such 
issues might be encountered by a lot of people anyway.

Could we try to fix it for those users? Either by filtering out 
"expensive" entries before saving (with some heuristic), or by entirely 
blacklisting command-history by default -- through 
savehist-ignored-variables.

Re: Turning on savehist-mode by default

[Eli Zaretskii]

> Date: Tue, 28 Nov 2023 15:07:46 -0600
> Cc: emacs-devel@gnu.org, public@thanosapollo.org
> From: Adam Porter <adam@alphapapa.net>
> 
> FWIW, IME, it may not be a good idea to enable savehist-mode by default. 
>   As an example, see this issue report on Ement.el's repository:
> 
> https://github.com/alphapapa/ement.el/issues/216
> 
> The gist is that, for users who enabled savehist, it was saving the 
> variable command-history, which included some complex commands whose 
> arguments (from interactive completion) included large data structures 
> (which are hidden from the user when called interactively).  This 
> resulted in savehist files hundreds of MB in size and very noticeable 
> pauses whenever savehist saved its data, from timers and upon Emacs exit.
> 
> The worst part was how long it took to discover the source of the 
> problem.  There were intermittent reports of weird pauses for months, 
> without any discernible reason, until a user finally narrowed it down to 
> using savehist.  Then we had to add this hacky workaround in Ement to 
> try to automatically avoid the problem.
> 
> Such cases may be rare (or not--I can't say), but when it happens, it 
> seems to be a bear to deal with, being very unintuitive.  For myself, 
> even having written a number of Emacs packages, I wasn't yet aware of 
> the command-history variable and how it saves arguments, and certainly 
> not the implications of using savehist as well.

savehist provides a customizable ability to limit the number of
history entries saved for each variable.

We could also add a new feature which limits the size of individual
entry: entries longer than that will not be saved.

Patches welcome.

Re: Turning on savehist-mode by default

[Björn Bidar]

Would undelete-frame-mode be a similar candidate for such a change?
I think so since it would improve the out of the box experience for most
users without bothering anyone else since the saved information doesn't
take up much memory and new users would have less issues when closing a
frame by accident.

Re: [OT] Not clobbering bash history

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I am trying to understand the words you wrote.

  > libhistory should, on recall attempt,

What does "recall attempt" mean?  I don't know that term.

                                          try to re-read the history file
  > from the last point it was on

What does "last point it was on" mean?

                                  in order to catch new histories, it should
  > append to the file,

What should it append, and in what circumstances?

                        and it should attempt to lock the file via flock or
  > similar if such facilities are available (just in case).

If two different shells will try to write history into one single file,
are they doomed to give bad results, one way or another>  It seems to me
that the crucial thing is for them to use two different files.

  > In the event of truncation, 

What truncates the file?  When does that occur?

Is such truncation _supposed_ to occur, or is it a bug that it occurs?

                                libhistory needs to be careful not to lose
  > any histories that were to be submitted in between the moment of
  > determination of truncation and commitment of the truncation to disk.

I don't follow.

  > As a QoL feature, bash should prevent history truncation if ran with
  > --norc or other flags that would inhibit HISTFILE being set potentially.

I don't understand this point -- can you explain?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [OT] Not clobbering bash history

[Arsen Arsenović]

[-- Attachment #1: Type: text/plain, Size: 3840 bytes --]

Hi,

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> I am trying to understand the words you wrote.
>
>   > libhistory should, on recall attempt,
>
> What does "recall attempt" mean?  I don't know that term.

I mean the action usually bound to C-p (I believe the libhistory
function is previous_history).

>                                           try to re-read the history file
>   > from the last point it was on
>
> What does "last point it was on" mean?

Say that the history has eight entries, and a ninth was added by a
parallel process, the current process would be at the point after the
eighth element and should continue reading from there next time it wants
a history entry.

>                                   in order to catch new histories, it should
>   > append to the file,
>
> What should it append, and in what circumstances?

New history entries, always (i.e. it should only ever append, as if via
O_APPEND, to ~/.bash_history and friends, at least while not holding
locks).

>                         and it should attempt to lock the file via flock or
>   > similar if such facilities are available (just in case).
>
> If two different shells will try to write history into one single file,
> are they doomed to give bad results, one way or another>  It seems to me
> that the crucial thing is for them to use two different files.

Not necessarily.  If both shells use a single write() syscall on an
O_APPEND file, they should work as expected to my awareness.  But, if
not, then locks would help (flock-ing the file before adding an entry
ensures only one shell is ever writing to enable correct history
storage).

>   > In the event of truncation, 
>
> What truncates the file?  When does that occur?

bash/libhistory would need to if it'd like to rotate the history file
due to HISTSIZE.

> Is such truncation _supposed_ to occur, or is it a bug that it occurs?

Depends.  If the user sets HISTSIZE to, say, 500, and enters a new
command, and it'd have been the 501st, the history file would need to be
truncated to be rotated.  In that scenario, it is intended.  If the user
did not intend to have HISTSIZE set to a positive value, then any
truncation is a bug (as history would be lost).

>                                 libhistory needs to be careful not to lose
>   > any histories that were to be submitted in between the moment of
>   > determination of truncation and commitment of the truncation to disk.
>
> I don't follow.

If a bash process decides to rotate the history file as a result of
HISTSIZE, and another bash process decides to do the same, one of their
new history entries would be lost due to the other one overriding it.
This would be a bug.

Hence, libhistory needs to be careful to prevent this.

>   > As a QoL feature, bash should prevent history truncation if ran with
>   > --norc or other flags that would inhibit HISTFILE being set potentially.
>
> I don't understand this point -- can you explain?

I meant to say HISTSIZE here.  My apologies.

If HISTSIZE is set in my RC file to a negative value (per (bash) Bash
Variables, this should inhibit history file rotation and/or truncation),
but if a shell is started with --norc (by, say, TRAMP, perhaps), then
bash would not be aware of the users desired HISTSIZE.  In that event,
it should be conservative and *not* overwrite the users' history.

I suspect that the latter is what was causing sporadic truncation in my
case (but I haven't confirmed so unfortunately).

Thanks, have a lovely day.
-- 
Arsen Arsenović

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 381 bytes --]

Re: [OT] Not clobbering bash history

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

I moved this to emacs-tangents because thus isn't really about Emacs,
but I don't know of a Bash list to use.  Is there one?

  > Say that the history has eight entries, and a ninth was added by a
  > parallel process, the current process would be at the point after the
  > eighth element and should continue reading from there next time it wants
  > a history entry.

Are you saying that multiple Bash processes in parallel should share
one single history, which would contain all the commands the user
entered in any of these processes?

So the commands of Basb process A and those of Basb process B would be
interspersed in the history?  And as you go forward and back in the
history in Bash process B, you would see A's commands and B's commands
in chronological order?

That makes a kind of sense, but what I would envision is that each
Bash process has its own history with only the commands of that process.

Why do you prefer the shared history file approach
to the one-history-per-process approach?
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)



---
via emacs-tangents mailing list (https://lists.gnu.org/mailman/listinfo/emacs-tangents)

Re: [OT] Not clobbering bash history

[mbork]

On 2023-12-06, at 21:49, Richard Stallman <rms@gnu.org> wrote:

> That makes a kind of sense, but what I would envision is that each
> Bash process has its own history with only the commands of that process.
>
> Why do you prefer the shared history file approach
> to the one-history-per-process approach?

Isn't it obvious?  If I have several terminals open at the same time
(and I seldom have fewer than, say, three, usually more), sharing
history is very useful.  It easy to remember that I issued some kind of
command, but much more difficult to remember in which terminal I did it.
Sharing history lessens my cognitive load.

Also, when I exit bash, the history is written to ~/.bash_history.  So,
if I have two bash sessions, they share the common history from that
file, but not common history from /this session/.  So the "every bash
process has a separate history" is not even true -- /part/ of the
history is shared.  Again, remembering which part adds to the cognitive
load.

Sharing history "live" makes it much simpler.

-- 
Marcin Borkowski
https://mbork.pl

---
via emacs-tangents mailing list (https://lists.gnu.org/mailman/listinfo/emacs-tangents)

Re: [OT] Not clobbering bash history

[Emanuel Berg]

mbork wrote:

>> That makes a kind of sense, but what I would envision is
>> that each Bash process has its own history with only the
>> commands of that process.
>>
>> Why do you prefer the shared history file approach to the
>> one-history-per-process approach?
>
> Isn't it obvious? If I have several terminals open at the
> same time (and I seldom have fewer than, say, three, usually
> more), sharing history is very useful. It easy to remember
> that I issued some kind of command, but much more difficult
> to remember in which terminal I did it. Sharing history
> lessens my cognitive load.

How do you access history? One item at a time?

Because if you do one task in one terminal, maybe vading
through history one item at a time will then be annoying since
items from everywhere else, from other tasks, will make you
have to cycle a lot before you find what you look for?

Not saying you are wrong, just the discussion can go both ways.
I'm not a history user myself, really, but my zsh history
isn't shared between tmux panes. Not in xterm, not in the
Linux console.

But I never gave it any thought until now.

-- 
underground experts united
https://dataswamp.org/~incal


---
via emacs-tangents mailing list (https://lists.gnu.org/mailman/listinfo/emacs-tangents)

Re: [OT] Not clobbering bash history

[Yuri Khan]

On Thu, 7 Dec 2023 at 12:40, <mbork@mbork.pl> wrote:

> Isn't it obvious?  If I have several terminals open at the same time
> (and I seldom have fewer than, say, three, usually more), sharing
> history is very useful.  It easy to remember that I issued some kind of
> command, but much more difficult to remember in which terminal I did it.

To add to this: The original terminal where I issued the command I
need might be busy, e.g. if I started some application in it. It might
be inconvenient to suspend that application just to get at the
command. Important special case: the application that’s occupying the
terminal where the original command was entered is Midnight Commander
running a sub-shell in which I want to repeat that command.

---
via emacs-tangents mailing list (https://lists.gnu.org/mailman/listinfo/emacs-tangents)

Re: [OT] Not clobbering bash history

[Emanuel Berg]

Yuri Khan wrote:

>> Isn't it obvious? If I have several terminals open at the
>> same time (and I seldom have fewer than, say, three,
>> usually more), sharing history is very useful. It easy to
>> remember that I issued some kind of command, but much more
>> difficult to remember in which terminal I did it.
>
> To add to this: The original terminal where I issued the
> command I need might be busy, e.g. if I started some
> application in it. It might be inconvenient to suspend that
> application just to get at the command. Important special
> case: the application that’s occupying the terminal where
> the original command was entered is Midnight Commander
> running a sub-shell in which I want to repeat that command.

Can you have both? Local history and global history.

You have a command to switch which one is at play. 'gh' for
global history, 'lh' for local and 'th' for toggle.

That way you would also see what you would use. It is possible
you would actually just use the global one as you say.

How do I get global history for zsh?

I have this file in play - it sounds like global history, but
maybe it is session local, then? So it should be "live"
between sessions as well? How do you get that?

#! bin/zsh
#
# this file:
#   https://dataswamp.org/~incal/conf/.zsh/history

setopt appendhistory
setopt histignorealldups

HISTSIZE=1000
SAVEHIST=$(( 10 * $HISTSIZE ))

HISTFILE=~/.zsh-history

-- 
underground experts united
https://dataswamp.org/~incal


---
via emacs-tangents mailing list (https://lists.gnu.org/mailman/listinfo/emacs-tangents)

Re: [OT] Not clobbering bash history

[Jean Louis]

>   > Say that the history has eight entries, and a ninth was added by a
>   > parallel process, the current process would be at the point after the
>   > eighth element and should continue reading from there next time it wants
>   > a history entry.

Arsen, explore options of the command `history' to do what you want,
you can save history:

$ history -a

and you can re-read history file:

$ history -r

and by using those options you can mix history from different Bash
sessions, and also re-use the history from other Bash session.



Jean

Take action in Free Software Foundation campaigns:
https://www.fsf.org/campaigns

In support of Richard M. Stallman
https://stallmansupport.org/

---
via emacs-tangents mailing list (https://lists.gnu.org/mailman/listinfo/emacs-tangents)

Re: [OT] Not clobbering bash history

[mbork]

On 2023-12-07, at 06:48, Emanuel Berg <incal@dataswamp.org> wrote:

> mbork wrote:
>
>>> That makes a kind of sense, but what I would envision is
>>> that each Bash process has its own history with only the
>>> commands of that process.
>>>
>>> Why do you prefer the shared history file approach to the
>>> one-history-per-process approach?
>>
>> Isn't it obvious? If I have several terminals open at the
>> same time (and I seldom have fewer than, say, three, usually
>> more), sharing history is very useful. It easy to remember
>> that I issued some kind of command, but much more difficult
>> to remember in which terminal I did it. Sharing history
>> lessens my cognitive load.
>
> How do you access history? One item at a time?

Either that, or C-r (i.e., searching), depending on the situation.

> Because if you do one task in one terminal, maybe vading
> through history one item at a time will then be annoying since
> items from everywhere else, from other tasks, will make you
> have to cycle a lot before you find what you look for?

Good point.

Best,

-- 
Marcin Borkowski
https://mbork.pl

---
via emacs-tangents mailing list (https://lists.gnu.org/mailman/listinfo/emacs-tangents)

Re: [OT] Not clobbering bash history

[Arsen Arsenović via Emacs news and miscellaneous discussions outside the scope of other Emacs mailing lists]

[-- Attachment #1.1: Type: text/plain, Size: 2131 bytes --]


Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
> I moved this to emacs-tangents because thus isn't really about Emacs,
> but I don't know of a Bash list to use.  Is there one?

IIRC there's bug-bash and it accepts patches, so it is perhaps also
appropriate for topics like this one.

>   > Say that the history has eight entries, and a ninth was added by a
>   > parallel process, the current process would be at the point after the
>   > eighth element and should continue reading from there next time it wants
>   > a history entry.
>
> Are you saying that multiple Bash processes in parallel should share
> one single history, which would contain all the commands the user
> entered in any of these processes?

Yes.  Opinions vary on this, but I have a workflow which relies on this
(via a hack I put in my bashrc and found ages ago).

> So the commands of Basb process A and those of Basb process B would be
> interspersed in the history?  And as you go forward and back in the
> history in Bash process B, you would see A's commands and B's commands
> in chronological order?

Yes.  ZSH does implement this well IME (but I haven't used ZSH
extensively.. so I'm not sure if I'd run into trouble with it).

> That makes a kind of sense, but what I would envision is that each
> Bash process has its own history with only the commands of that process.

Some do prefer that.  It'd be reasonable to do that by default, as it's
currently the default.

> Why do you prefer the shared history file approach
> to the one-history-per-process approach?

Because I rely on working in many, many interactive shells in parallel.

Though, on that note, it is perhaps reasonable to flip 'histappend' to
on by default, even if nothing else changes (because otherwise history
is lost since only the last shells history is kept).

Thanks, have a lovely day.
-- 
Arsen Arsenović

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[-- Attachment #2: Type: text/plain, Size: 92 bytes --]

---
via emacs-tangents mailing list (https://lists.gnu.org/mailman/listinfo/emacs-tangents)

Re: [OT] Not clobbering bash history

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Since this is about Bash, ot Emacs, I moved it to emacs-tangents@gnu.org.
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [OT] Not clobbering bash history

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > If two different shells will try to write history into one single file,
  > > are they doomed to give bad results, one way or another...

  > Not necessarily.  If both shells use a single write() syscall on an
  > O_APPEND file, they should work as expected to my awareness.

We are miscommunicating.  The way you expect it to work is, in my
opinion, a bad result -- various histories interspersed.

It seems to me that the crucial thing is for each Bash process
to have its own separate history.

Do you think that behavior would be bad?

  > If a bash process decides to rotate the history file as a result of
  > HISTSIZE, and another bash process decides to do the same, one of their
  > new history entries would be lost due to the other one overriding it.
  > This would be a bug.

Only if they share one single history file.  If each has its own
history file, each can handle it as if it were your only Bash process.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)



---
via emacs-tangents mailing list (https://lists.gnu.org/mailman/listinfo/emacs-tangents)

Re: [OT] Not clobbering bash history

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > So the commands of Basb process A and those of Basb process B would be
  > > interspersed in the history?  And as you go forward and back in the
  > > history in Bash process B, you would see A's commands and B's commands
  > > in chronological order?

  > Yes.  ZSH does implement this well IME (but I haven't used ZSH
  > extensively.. so I'm not sure if I'd run into trouble with it).

  > > That makes a kind of sense, but what I would envision is that each
  > > Bash process has its own history with only the commands of that process.

  > Some do prefer that.  It'd be reasonable to do that by default, as it's
  > currently the default.

It seems fine to support both modes and let each user choose.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)



---
via emacs-tangents mailing list (https://lists.gnu.org/mailman/listinfo/emacs-tangents)

Re: [OT] Not clobbering bash history

[brickviking]

[-- Attachment #1.1: Type: text/plain, Size: 3034 bytes --]

Please continue to CC: me on this thread, as I'm no longer part of the
emacs-devel
mailing list for the moment.

On Fri, 8 Dec 2023 at 16:54, Richard Stallman <rms@gnu.org> wrote:

>   > > If two different shells will try to write history into one single
> file,
>   > > are they doomed to give bad results, one way or another...
>
>   > Not necessarily.  If both shells use a single write() syscall on an
>   > O_APPEND file, they should work as expected to my awareness.
>
> We are miscommunicating.  The way you expect it to work is, in my
> opinion, a bad result -- various histories interspersed.
>
> It seems to me that the crucial thing is for each Bash process
> to have its own separate history.
>
> Do you think that behavior would be bad?
>
>
In my opinion, if I wanted to search for a command I'd previously run that
wasn't in my current bash shell history, how would the respective histories
be created?
Would it involve .bash_history.${PID},  or would it involve
.bash_history.${CURRENT_TIMESTAMP} (whatever form that took),
or some other combination?

In each case, bash can already look up its own history, but when a new Bash
shell is
started, just which history file is it going to load if there are multiple
bash history files
saved to disk? The latest by ${CURRENT_TIMESTAMP}? The highest ${PID}?
And where will subsequent history be written to? A new ${PID} file?

People who run multiple shells in parallel are probably well aware of this
now, and I expect that is why decisions were made way back when, to support
one history file. The only real argument is whether that history gets
erased when a current bash shell closes out and saves its own history over
the top (sort of handled by histappend, but not entirely.)

  > If a bash process decides to rotate the history file as a result of
>   > HISTSIZE, and another bash process decides to do the same, one of their
>   > new history entries would be lost due to the other one overriding it.
>   > This would be a bug.
>
> Only if they share one single history file.  If each has its own
> history file, each can handle it as if it were your only Bash process.
>
>
I must admit to having thought that histories could be stored in a sqlite3
database,
that way rotation of expired entries (date or number of lines) could be
handled by the
database itself. However, this means yet another dependency added to the
chain
where you might not have wished one. And in addition, the licence that
sqlite3 is provided
under is not a GNU licence, it's one of those licences that pretty much says
"We donate this to the public at large" (my summary, not theirs).

Another disadvantage for sqlite as a dependency, is that it's not the
simplest solution
feasible. It is darn quick though, and its development is certainly current
and ongoing.
One advantage to making the history as a single file (flat or database) is
to make
searching easier, but that's also its Achilles heel when you want to rotate
the file if you're running
multiple bash shells.

Regards, brickviking

[-- Attachment #1.2: Type: text/html, Size: 4023 bytes --]

[-- Attachment #2: Type: text/plain, Size: 92 bytes --]

---
via emacs-tangents mailing list (https://lists.gnu.org/mailman/listinfo/emacs-tangents)

Re: [OT] Not clobbering bash history

[Arsen Arsenović via Emacs news and miscellaneous discussions outside the scope of other Emacs mailing lists]

[-- Attachment #1.1: Type: text/plain, Size: 1646 bytes --]


Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > If two different shells will try to write history into one single file,
>   > > are they doomed to give bad results, one way or another...
>
>   > Not necessarily.  If both shells use a single write() syscall on an
>   > O_APPEND file, they should work as expected to my awareness.
>
> We are miscommunicating.  The way you expect it to work is, in my
> opinion, a bad result -- various histories interspersed.
>
> It seems to me that the crucial thing is for each Bash process
> to have its own separate history.
>
> Do you think that behavior would be bad?

Bad?  No.  It's not what I'd prefer, though.

Note that, with either separate or interspersed histories, history
should never be lost, so, if a file is being shared by multiple shells
(even if it is not continuously re-read), care should be taken not to
lose data.

>   > If a bash process decides to rotate the history file as a result of
>   > HISTSIZE, and another bash process decides to do the same, one of their
>   > new history entries would be lost due to the other one overriding it.
>   > This would be a bug.
>
> Only if they share one single history file.  If each has its own
> history file, each can handle it as if it were your only Bash process.

Indeed.  How would these histories be recalled, though?  Which file does
a new shell read?
-- 
Arsen Arsenović

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 381 bytes --]

[-- Attachment #2: Type: text/plain, Size: 92 bytes --]

---
via emacs-tangents mailing list (https://lists.gnu.org/mailman/listinfo/emacs-tangents)

Re: [OT] Not clobbering bash history

[Arsen Arsenović via Emacs news and miscellaneous discussions outside the scope of other Emacs mailing lists]

[-- Attachment #1.1: Type: text/plain, Size: 1059 bytes --]


Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > So the commands of Basb process A and those of Basb process B would be
>   > > interspersed in the history?  And as you go forward and back in the
>   > > history in Bash process B, you would see A's commands and B's commands
>   > > in chronological order?
>
>   > Yes.  ZSH does implement this well IME (but I haven't used ZSH
>   > extensively.. so I'm not sure if I'd run into trouble with it).
>
>   > > That makes a kind of sense, but what I would envision is that each
>   > > Bash process has its own history with only the commands of that process.
>
>   > Some do prefer that.  It'd be reasonable to do that by default, as it's
>   > currently the default.
>
> It seems fine to support both modes and let each user choose.

I agree.
-- 
Arsen Arsenović

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 381 bytes --]

[-- Attachment #2: Type: text/plain, Size: 92 bytes --]

---
via emacs-tangents mailing list (https://lists.gnu.org/mailman/listinfo/emacs-tangents)

Re: [OT] Not clobbering bash history

[Arsen Arsenović via Emacs news and miscellaneous discussions outside the scope of other Emacs mailing lists]

[-- Attachment #1.1: Type: text/plain, Size: 1093 bytes --]

Hi Jean,

Jean Louis <bugs@gnu.support> writes:

>>   > Say that the history has eight entries, and a ninth was added by a
>>   > parallel process, the current process would be at the point after the
>>   > eighth element and should continue reading from there next time it wants
>>   > a history entry.
>
> Arsen, explore options of the command `history' to do what you want,
> you can save history:
>
> $ history -a
>
> and you can re-read history file:
>
> $ history -r
>
> and by using those options you can mix history from different Bash
> sessions, and also re-use the history from other Bash session.

Thanks for the tip.  I already have something using these flags
implemented:

  PROMPT_COMMAND="$PROMPT_COMMAND;history -a; history -c; history -r")

which is a functional hack, for the most part.  It still has some of the
problems mentioned in my original message (history is only added to
after a command concludes, sporadic truncation still happens, etc), plus
this approach is rather ugly.

Thanks again, have a lovely day.
-- 
Arsen Arsenović

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 381 bytes --]

[-- Attachment #2: Type: text/plain, Size: 92 bytes --]

---
via emacs-tangents mailing list (https://lists.gnu.org/mailman/listinfo/emacs-tangents)

Re: Turning on savehist-mode by default

[Stefan Kangas]

Po Lu <luangruo@yahoo.com> writes:

> Emacs will behave inconsistently, subject to whether it's been killed
> such that the hook cannot run.  And the latter runs the risk of files
> being corrupted if the file is being saved, by chance, the instant
> Emacs is killed.

That just sounds like a bug in the hook function.

I think we could fix it by creating the file separately and then moving
it into place.

Re: Turning on savehist-mode by default

[Stefan Kangas]

sbaugh@catern.com writes:

> savehist-mode is a useful mode which is turned on by many Emacs
> users. [1] It matches the default behavior of programs like bash and
> vim, which save command history by default.  I suggest that we should
> find some way to enable savehist by default.

Yes, good idea.  This functionality seems to be expected in modern
software, so I don't see why we shouldn't provide it.

I saw some discussion about desktop.el as an alternative.  To me that's
a separate discussion.  (FWIW, I'd rather see that we unified
savehist-mode and desktop.el instead of maintaining duplicate
functionality.  I have no idea if that can be done in a reasonable way.)

I didn't see any serious objections, so perhaps we could move forward
with this proposal.  Eli, WDYT?

> By default savehist creates a timer which saves the history every 5
> minutes.  I suggest that by default we should just have behavior like:
> (add-hook 'kill-emacs-hook #'savehist-autosave)
> If the user explicitly runs (savehist-mode 1) then that would enable the
> timer-based saving.

Having only the `kill-emacs-hook' seems like a good step forward, but
why not have both the timer and the hook?  Otherwise, if Emacs crashes,
the user will lose her command history.  That's not what we want, I
think.

BTW, and this is a bit of a tangent, but the "every 5 minutes" default
could probably safely be lowered a little bit (1-3 minutes or so).

Re: Turning on savehist-mode by default

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Sat, 16 Dec 2023 11:01:22 -0800
> 
> sbaugh@catern.com writes:
> 
> > savehist-mode is a useful mode which is turned on by many Emacs
> > users. [1] It matches the default behavior of programs like bash and
> > vim, which save command history by default.  I suggest that we should
> > find some way to enable savehist by default.
> 
> Yes, good idea.  This functionality seems to be expected in modern
> software, so I don't see why we shouldn't provide it.
> 
> I saw some discussion about desktop.el as an alternative.  To me that's
> a separate discussion.  (FWIW, I'd rather see that we unified
> savehist-mode and desktop.el instead of maintaining duplicate
> functionality.  I have no idea if that can be done in a reasonable way.)

(FWIW, if savehist-mode will be on by default, I will turn it off in
my init file, because I already use desktop.el to save the few history
variables I care about.)

> I didn't see any serious objections, so perhaps we could move forward
> with this proposal.  Eli, WDYT?

There wasn't a compelling reason to turn it on, either.  "Other
programs save history" is not enough in my book.  Other programs also
have the equivalent of recentf ON by default, but we don't.

On disadvantage that was never discussed is that if we turn it on by
default, we'd need to preload savehist.

All in all, I'm not sure we should do this.  Perhaps a more wide user
poll is in order.

Re: Turning on savehist-mode by default

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Kangas <stefankangas@gmail.com>
>> Date: Sat, 16 Dec 2023 11:01:22 -0800
>>
>> I didn't see any serious objections, so perhaps we could move forward
>> with this proposal.  Eli, WDYT?
>
> There wasn't a compelling reason to turn it on, either.  "Other
> programs save history" is not enough in my book.  Other programs also
> have the equivalent of recentf ON by default, but we don't.

This is my reasoning:

Users these days will expect that their history is saved between
invocations of Emacs, as most other programs do that (including
e.g. bash and even many languages REPLs).  This customization is common
in many starter kits, and is frequently recommended to beginners.  Not
doing it is somewhat surprising and inconvenient.

Meanwhile, I see few drawbacks to enabling it.

As for `recentf', I think it's a bit different from `savehist-mode', as
it's only used in particular use cases.  I can only think of: mousing in
the menu bar and calling `M-x recentf-open'.  On the other hand,
`savehist-mode' affects anyone that uses the minibuffer and knows to
press `M-p'.

> On disadvantage that was never discussed is that if we turn it on by
> default, we'd need to preload savehist.

I didn't check how it affects the size of a running Emacs.  The ELC is
only around 12 KB here, FWIW.

> All in all, I'm not sure we should do this.  Perhaps a more wide user
> poll is in order.

I suppose we could ask people on /r/emacs, if that would help.

Re: Turning on savehist-mode by default

[Po Lu]

Stefan Kangas <stefankangas@gmail.com> writes:

> Meanwhile, I see few drawbacks to enabling it.

Multiple, concurrent Emacs sessions will cease behaving in an orderly
and predictable manner.  It makes little sense for Emacs to provide
features designed to enable such orderly operation, such as lock files,
while undermining them by enabling recentf or savehist by default, both
of which overwrite files other sessions might've generated with their
own.

Re: Turning on savehist-mode by default

[Adam Porter]

In addition to what Po said, I'd like to gently reiterate what I said 
earlier in this thread: Given my experience with savehist causing 
unexpected and hard-to-debug performance problems[0], I'd guess that 
there are more such cases in the wild waiting to be triggered.  If, e.g. 
Emacs 30.1 enabled it by default, I can imagine a number of users 
suddenly encountering weird pauses, and they'd probably blame GC 
initially[1].

As well, I have some concerns about savehist's having the potential to 
cause weird bugs in other libraries: The savehist-save function seems to 
comment out individual elements of savehist-minibuffer-history-variables 
that it determines are unreadable.  That's understandable from its 
perspective, but what effect will that have on libraries that may not be 
expecting for their data structures to have certain parts disappear 
after restarting Emacs?  I can just imagine the bug reports from users 
showing apparently corrupted or elided data structures, and having no 
clue as to what is mutating them, because the code isn't within the 
library having a bug reported against it.

IMHO, a feature like this is indeed expected and overdue in Emacs, but I 
don't think savehist is a good way to enable it, given its current 
implementation.  ISTM that something built on top of the new 
multisession library would be more appropriate and (at least, have the 
potential to be) more robust.

If savehist is to be used for this, I would at least recommend taking a 
release cycle to publicly ask users to enable it by default and report 
the results, and wait for at least a few hundred positive reports before 
enabling it by default.  Otherwise, I would expect not-insignificant 
fallout, IMHO.

Thanks,
Adam

0: https://github.com/alphapapa/ement.el/issues/216

1: Which may, in fact, be part of the problem, considering how 
savehist-save works: it prints a Lisp object to a buffer, then tries to 
read it to determine whether it's readable--doesn't that generate 
garbage?  Now consider doing that every 1-5 minutes and upon exit.

Re: Turning on savehist-mode by default

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Sat, 16 Dec 2023 14:57:48 -0800
> Cc: sbaugh@catern.com, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Stefan Kangas <stefankangas@gmail.com>
> >> Date: Sat, 16 Dec 2023 11:01:22 -0800
> >>
> >> I didn't see any serious objections, so perhaps we could move forward
> >> with this proposal.  Eli, WDYT?
> >
> > There wasn't a compelling reason to turn it on, either.  "Other
> > programs save history" is not enough in my book.  Other programs also
> > have the equivalent of recentf ON by default, but we don't.
> 
> This is my reasoning:
> 
> Users these days will expect that their history is saved between
> invocations of Emacs, as most other programs do that (including
> e.g. bash and even many languages REPLs).  This customization is common
> in many starter kits, and is frequently recommended to beginners.  Not
> doing it is somewhat surprising and inconvenient.

Whether users indeed expect this is something I suggest to verify, by
polling.  What is the percentage of users who enable savehist in their
customizations?

> > One disadvantage that was never discussed is that if we turn it on by
> > default, we'd need to preload savehist.
> 
> I didn't check how it affects the size of a running Emacs.  The ELC is
> only around 12 KB here, FWIW.

Well, yes, it is not huge, but we shouldn't preload anything we don't
have to, because it's a penalty for everyone, even those who turn the
feature off.

> > All in all, I'm not sure we should do this.  Perhaps a more wide user
> > poll is in order.
> 
> I suppose we could ask people on /r/emacs, if that would help.

Yes, I think we should.  But this should be organized so that we will
know the fraction of users who turn this on, so we'd need to hear from
people who turn it on and from those who don't.

Re: Turning on savehist-mode by default

[Stefan Kangas]

Adam Porter <adam@alphapapa.net> writes:

> In addition to what Po said, I'd like to gently reiterate what I said
> earlier in this thread: Given my experience with savehist causing
> unexpected and hard-to-debug performance problems[0], I'd guess that
> there are more such cases in the wild waiting to be triggered.  If, e.g.
> Emacs 30.1 enabled it by default, I can imagine a number of users
> suddenly encountering weird pauses, and they'd probably blame GC
> initially[1].
>
> As well, I have some concerns about savehist's having the potential to
> cause weird bugs in other libraries: The savehist-save function seems to
> comment out individual elements of savehist-minibuffer-history-variables
> that it determines are unreadable.  That's understandable from its
> perspective, but what effect will that have on libraries that may not be
> expecting for their data structures to have certain parts disappear
> after restarting Emacs?  I can just imagine the bug reports from users
> showing apparently corrupted or elided data structures, and having no
> clue as to what is mutating them, because the code isn't within the
> library having a bug reported against it.
>
> IMHO, a feature like this is indeed expected and overdue in Emacs, but I
> don't think savehist is a good way to enable it, given its current
> implementation.  ISTM that something built on top of the new
> multisession library would be more appropriate and (at least, have the
> potential to be) more robust.

Thanks, this is clear and compelling.

It seems to me that savehist-mode is *not* ready for being enabled by
default.  I think the best we can do at this point is to start working
on improve and/or replace it.

Would anyone like to volunteer to do that?

> 0: https://github.com/alphapapa/ement.el/issues/216

Did you think about how issues like the above could be solved (in
general, not necessarily in savehist specifically)?

> 1: Which may, in fact, be part of the problem, considering how
> savehist-save works: it prints a Lisp object to a buffer, then tries to
> read it to determine whether it's readable--doesn't that generate
> garbage?  Now consider doing that every 1-5 minutes and upon exit.

savehist (or it's replacement) could also be smarter about it, for
example you could skip re-serializing history variables that have not
been touched.

Re: Turning on savehist-mode by default

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  sbaugh@catern.com,  emacs-devel@gnu.org
> Date: Sun, 17 Dec 2023 07:57:07 +0800
> 
> Stefan Kangas <stefankangas@gmail.com> writes:
> 
> > Meanwhile, I see few drawbacks to enabling it.
> 
> Multiple, concurrent Emacs sessions will cease behaving in an orderly
> and predictable manner.  It makes little sense for Emacs to provide
> features designed to enable such orderly operation, such as lock files,
> while undermining them by enabling recentf or savehist by default, both
> of which overwrite files other sessions might've generated with their
> own.

Isn't this a general issue with saving any customizations?  E.g.,
saving options, saving desktop, etc. etc.?  Saving history from Bash,
GDB, etc. has the same problems, AFAIK.

And anyway, the problem already exists for those who turn on savehist
in their customizations.  If your usage patterns must avoid the
overwriting you mention above, simply disable savehist in your init
file, and Bob's your uncle.

IOW, once again: personal preferences should NOT affect the defaults,
nor are they too important when discussing those defaults.  Unless the
feature we want to turn on is buggy (in which case it must be fixed
first), turning it on by default cannot possibly cause problems we
don't already have.

Re: Turning on savehist-mode by default

[Eli Zaretskii]

> Date: Sat, 16 Dec 2023 23:42:15 -0600
> Cc: eliz@gnu.org, emacs-devel@gnu.org, sbaugh@catern.com,
>  stefankangas@gmail.com
> From: Adam Porter <adam@alphapapa.net>
> 
> In addition to what Po said, I'd like to gently reiterate what I said 
> earlier in this thread: Given my experience with savehist causing 
> unexpected and hard-to-debug performance problems[0]

That mentions a single 3rd-party package that triggers the issue, and
includes a workaround solution.  I see nothing awful there.

> I'd guess that there are more such cases in the wild waiting to be
> triggered.

You know about other packages that add huge elements to history
variables?  Which ones?

> If, e.g.  Emacs 30.1 enabled it by default, I can imagine a number
> of users suddenly encountering weird pauses, and they'd probably
> blame GC initially[1].

This remains to be seen, and having this on master enough time in
advance will allow such reports to come in, if indeed such problems
exist.

We could also introduce a limitation of element size (a defcustom) to
be imposed by add-to-history, if we think such long elements are
detrimental to performance.

> As well, I have some concerns about savehist's having the potential to 
> cause weird bugs in other libraries: The savehist-save function seems to 
> comment out individual elements of savehist-minibuffer-history-variables 
> that it determines are unreadable.  That's understandable from its 
> perspective, but what effect will that have on libraries that may not be 
> expecting for their data structures to have certain parts disappear 
> after restarting Emacs?  I can just imagine the bug reports from users 
> showing apparently corrupted or elided data structures, and having no 
> clue as to what is mutating them, because the code isn't within the 
> library having a bug reported against it.

If such bug reports will come in, we will handle them.  As we do with
any other Emacs feature.  Why worry in advance, when we don't yet have
any such reports, and therefore can do nothing about them?

> IMHO, a feature like this is indeed expected and overdue in Emacs, but I 
> don't think savehist is a good way to enable it, given its current 
> implementation.  ISTM that something built on top of the new 
> multisession library would be more appropriate and (at least, have the 
> potential to be) more robust.
> 
> If savehist is to be used for this, I would at least recommend taking a 
> release cycle to publicly ask users to enable it by default and report 
> the results, and wait for at least a few hundred positive reports before 
> enabling it by default.  Otherwise, I would expect not-insignificant 
> fallout, IMHO.

If we enable it on master, it will give users precisely that, right?

Re: Turning on savehist-mode by default

[tomas]

[-- Attachment #1: Type: text/plain, Size: 519 bytes --]

On Sun, Dec 17, 2023 at 09:40:32AM +0200, Eli Zaretskii wrote:
> > From: Stefan Kangas <stefankangas@gmail.com>

[...]

> > Users these days will expect [...]

> Whether users indeed expect this is something I suggest to verify [...]

I always flinch a bit when I read this. Especially in the context
of Emacs, which arguably has the most heterogeneous user base any
piece of software might have these days.

Which makes your (the maintainer's) job all that more astonishing :)

Cheers & thanks
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Turning on savehist-mode by default

[Adam Porter]

On 12/17/23 01:49, Stefan Kangas wrote:

>> 0: https://github.com/alphapapa/ement.el/issues/216
> 
> Did you think about how issues like the above could be solved (in
> general, not necessarily in savehist specifically)?

It doesn't seem like an easy problem.  My guess as to the most "correct" 
solution would be to limit how much time can be spent serializing the 
data, both as a whole and individual elements, and to cancel or 
interrupt the serialization of the whole or a part if the limit is 
exceeded--the goal being to avoid UI pauses.  Then the limit could be 
disabled when desired (e.g. when exiting Emacs, wanting to be sure the 
data is saved).  But doing that would require some bookkeeping in the 
background, which seems like it might be excessive complexity for what 
should be a simple feature.

Re: Turning on savehist-mode by default

[Adam Porter]

On 12/17/23 02:12, Eli Zaretskii wrote:
>> Date: Sat, 16 Dec 2023 23:42:15 -0600
>> Cc: eliz@gnu.org, emacs-devel@gnu.org, sbaugh@catern.com,
>>   stefankangas@gmail.com
>> From: Adam Porter <adam@alphapapa.net>
>>
>> In addition to what Po said, I'd like to gently reiterate what I said
>> earlier in this thread: Given my experience with savehist causing
>> unexpected and hard-to-debug performance problems[0]
> 
> That mentions a single 3rd-party package that triggers the issue, and
> includes a workaround solution.  I see nothing awful there.

 From my perspective, as the developer of the package being accused of 
having performance problems by random users, problems I couldn't 
reproduce, nor even begin to guess what the cause was, or if it was even 
an actual problem, it felt pretty awful.

>> I'd guess that there are more such cases in the wild waiting to be
>> triggered.
> 
> You know about other packages that add huge elements to history
> variables?  Which ones?

My package does not add huge elements to history variables.  It simply 
passes arguments to functions via their interactive forms.  I didn't 
even know that such history variables existed until that bug report came 
to its conclusion, and I've used Emacs for years and published tens of 
packages which together have nearly a million downloads.  So if it can 
happen to me, it can probably happen to anyone.

>> If, e.g.  Emacs 30.1 enabled it by default, I can imagine a number
>> of users suddenly encountering weird pauses, and they'd probably
>> blame GC initially[1].
> 
> This remains to be seen, and having this on master enough time in
> advance will allow such reports to come in, if indeed such problems
> exist.

You may be right, considering how many users who build master have been 
speaking up lately.  Still, my concern is that whether the problem 
happens depends very much on a user's established workflows and 
installed packages.  Some users will never encounter the problem, and 
other users will encounter it constantly, and those who do won't have a 
clue what's going on, because it's heavily obscured.  I don't know what 
sample size would be needed to be likely to detect such problems, but it 
wouldn't be small, given that probably 0.1% or less of the users of 
Ement.el seemed to experience the problem--but for those who did, it was 
crippling to their Emacs usage.

> We could also introduce a limitation of element size (a defcustom) to
> be imposed by add-to-history, if we think such long elements are
> detrimental to performance.

That would likely be necessary, yes.  No one wants 450 MB savehist files 
in their .emacs.d getting rewritten every few minutes.

>> As well, I have some concerns about savehist's having the potential to
>> cause weird bugs in other libraries: The savehist-save function seems to
>> comment out individual elements of savehist-minibuffer-history-variables
>> that it determines are unreadable.  That's understandable from its
>> perspective, but what effect will that have on libraries that may not be
>> expecting for their data structures to have certain parts disappear
>> after restarting Emacs?  I can just imagine the bug reports from users
>> showing apparently corrupted or elided data structures, and having no
>> clue as to what is mutating them, because the code isn't within the
>> library having a bug reported against it.
> 
> If such bug reports will come in, we will handle them.  As we do with
> any other Emacs feature.  Why worry in advance, when we don't yet have
> any such reports, and therefore can do nothing about them?

I guess it's a matter of perspective--from mine, I'm sharing such a report.

Re: Turning on savehist-mode by default

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Isn't this a general issue with saving any customizations?  E.g.,
> saving options, saving desktop, etc. etc.?  Saving history from Bash,
> GDB, etc. has the same problems, AFAIK.

Yes, and that's one reason I'm glad Emacs isn't Bash!

> And anyway, the problem already exists for those who turn on savehist
> in their customizations.  If your usage patterns must avoid the
> overwriting you mention above, simply disable savehist in your init
> file, and Bob's your uncle.

What about emacs -q, or if an error is signaled by savehist itself on
loading a malformed savehist file, or any other of the myriad of
circumstances that might cause Emacs to start without completely loading
the user initialization file?

> IOW, once again: personal preferences should NOT affect the defaults,
> nor are they too important when discussing those defaults.  Unless the
> feature we want to turn on is buggy (in which case it must be fixed
> first), turning it on by default cannot possibly cause problems we
> don't already have.

These aren't personal preferences.  Running multiple sessions at once is
a feature which Emacs has long supported six ways from Sunday, and it
should continue to work by default.

Hence we should resolve this problem with savehist in advance, if only
to avoid confusing users who do run Emacs in this manner.  This is by no
means all that is necessary to enable savehist-mode by default, of
course; savehist establishes the location of the savehist file when it
is loaded, and loads its history file upon the major mode being enabled,
so this change will certainly be more involved than loading and enabling
savehist-mode during loadup.

Thus with the aforementioned disadvantages on one hand, and all the toil
entailed on the other, is it any wonder that some people are averse to
this proposal?

Just my two cents, thanks.

Re: Turning on savehist-mode by default

[Adam Porter]

[-- Attachment #1: Type: text/plain, Size: 569 bytes --]

On 12/17/23 01:49, Stefan Kangas wrote:
> It seems to me that savehist-mode is*not*  ready for being enabled by
> default.  I think the best we can do at this point is to start working
> on improve and/or replace it.
> 
> Would anyone like to volunteer to do that?
> 
>> 0:https://github.com/alphapapa/ement.el/issues/216
> Did you think about how issues like the above could be solved (in
> general, not necessarily in savehist specifically)?

FWIW, I hacked this up, and it's very basic, but it seems to work (very 
lightly tested).  Might be a useful starting point.

[-- Attachment #2: varhist.el --]
[-- Type: text/x-emacs-lisp, Size: 4139 bytes --]

;;; varhist.el --- Persist variables' values across sessions  -*- lexical-binding: t; -*-

;; Copyright (C) 2023  Free Software Foundation, Inc.

;; Author: Adam Porter <adam@alphapapa.net>
;; Keywords: convenience, convenience

;; This program is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.

;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with this program.  If not, see <https://www.gnu.org/licenses/>.

;;; Commentary:

;; 

;;; Code:

(require 'cl-lib)
(require 'multisession)

(define-multisession-variable varhist-values (make-hash-table)
  "Table storing variables and values."
  :synchronized t)

(defvar varhist-timer nil)

(defgroup varhist nil
  "Persist the history of certain variables across sessions."
  :group 'convenience)

(defcustom varhist-variables nil
  "Vars to save."
  :type '(repeat symbol))

(defcustom varhist-timeout 0.2
  "Timeout in seconds for saving all variables."
  :type 'natnum)

(defcustom varhist-repeat "5 minutes"
  "Save values during a session this often.
A value accepted by `run-at-time', which see."
  :type '(choice (string :tag "A relative time string")
                 (natnum :tag "Number of seconds")
                 (const :tag "Don't save during a session" nil)))

(defcustom varhist-quietly t
  "Load and save without displaying messages."
  :type 'boolean)

(define-minor-mode varhist-mode
  "Persist the history of certain variables across sessions.
Loads values for variables in `varhist-variables' when activated;
saves values when deactivated."
  :global t
  (if varhist-mode
      (progn
        (add-hook 'kill-emacs-hook #'varhist--killing-emacs)
        (when varhist-repeat
          (setf varhist-timer
                (run-at-time varhist-repeat
                             (cl-typecase varhist-repeat
                               (number varhist-repeat)
                               (string (timer-duration varhist-repeat)))
                             #'varhist--save)))
        (varhist--load))
    (remove-hook 'kill-emacs-hook #'varhist--killing-emacs)
    (when (timerp varhist-timer)
      (cancel-timer varhist-timer)
      (setf varhist-timer nil))
    (varhist--save)))

(defun varhist--killing-emacs ()
  "Call `varhist--save' safely.
Ignores errors and uses a long timeout.  Safe for calling from
`kill-emacs-hook'."
  (ignore-errors
    ;; Use a long timeout to ensure values are saved when exiting.
    (varhist--save nil 10)))

(defun varhist--save (&optional variables timeout)
  ;; First, reload the values table in case another session has
  ;; changed some of them (even though we will overwrite them, the
  ;; other session might include variables we don't, so we should
  ;; re-read it first).
  (let ((values (multisession-value varhist-values))
        (variables (or variables varhist-variables))
        (timeout (or timeout varhist-timeout)))
    (with-timeout
        (timeout
         (display-warning 'varhist "Saving timed out"))
      (dolist (variable variables)
        (setf (gethash variable values) (symbol-value variable))
        (sit-for 0 'nodisp))
      (setf (multisession-value varhist-values) values))
    (unless varhist-quietly
      (message "varhist: Saved."))))

(defun varhist--load (&optional variables)
  (let ((values (multisession-value varhist-values))
        (variables (or variables varhist-variables)))
    (dolist (variable variables)
      ;; Only set variables present in the table.
      (pcase (gethash variable values 'varhist-not-found)
        ('varhist-not-found)
        (value (setf (symbol-value variable) value))))
    (unless varhist-quietly
      (message "varhist: Loaded."))))

(provide 'varhist)
;;; varhist.el ends here

Re: Turning on savehist-mode by default

[Eli Zaretskii]

> Date: Sun, 17 Dec 2023 05:19:31 -0600
> Cc: luangruo@yahoo.com, emacs-devel@gnu.org, sbaugh@catern.com,
>  stefankangas@gmail.com
> From: Adam Porter <adam@alphapapa.net>
> 
> On 12/17/23 02:12, Eli Zaretskii wrote:
> > 
> > That mentions a single 3rd-party package that triggers the issue, and
> > includes a workaround solution.  I see nothing awful there.
> 
>  From my perspective, as the developer of the package being accused of 
> having performance problems by random users, problems I couldn't 
> reproduce, nor even begin to guess what the cause was, or if it was even 
> an actual problem, it felt pretty awful.

Welcome to the club.  I (and other maintainers) get that feeling
almost every day.  And yet I don't conclude from this that existing
features are badly implemented or that they should be removed or
turned off by default.  Instead, we investigate the issues, find the
problems, and either fix them or provide a workaround.

So I see nothing special in the incident you referred to: it's what I
see every day.

> >> I'd guess that there are more such cases in the wild waiting to be
> >> triggered.
> > 
> > You know about other packages that add huge elements to history
> > variables?  Which ones?
> 
> My package does not add huge elements to history variables.  It simply 
> passes arguments to functions via their interactive forms.  I didn't 
> even know that such history variables existed until that bug report came 
> to its conclusion, and I've used Emacs for years and published tens of 
> packages which together have nearly a million downloads.  So if it can 
> happen to me, it can probably happen to anyone.

That was not my question, sorry.  My question was whether you are
actually _aware_ of such problems happening in other cases with other
packages.  Because if not, a single instance is not yet a reason for
considering this a grave problem, a reason serious enough to consider
savehist flawed or immature.

Re: Turning on savehist-mode by default

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: stefankangas@gmail.com,  sbaugh@catern.com,  emacs-devel@gnu.org
> Date: Sun, 17 Dec 2023 19:48:02 +0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > And anyway, the problem already exists for those who turn on savehist
> > in their customizations.  If your usage patterns must avoid the
> > overwriting you mention above, simply disable savehist in your init
> > file, and Bob's your uncle.
> 
> What about emacs -q, or if an error is signaled by savehist itself on
> loading a malformed savehist file, or any other of the myriad of
> circumstances that might cause Emacs to start without completely loading
> the user initialization file?

Turning off savehist-mode will fix all of those, no?

You are basically describing something that can happen with any Emacs
feature that is turned on by default.  The solution is to turn them
off when you want to avoid any of their effects.  I routinely do that
with blink-cursor-mode, global-eldoc-mode, and sometimes others, when
they might get in the way.  Or invoke "emacs -D".

> > IOW, once again: personal preferences should NOT affect the defaults,
> > nor are they too important when discussing those defaults.  Unless the
> > feature we want to turn on is buggy (in which case it must be fixed
> > first), turning it on by default cannot possibly cause problems we
> > don't already have.
> 
> These aren't personal preferences.  Running multiple sessions at once is
> a feature which Emacs has long supported six ways from Sunday, and it
> should continue to work by default.

That's true, but using such multiple sessions (as opposed to a single
session) is a personal preference.  Resisting some change in the
defaults because it adversely affects you personally is not a valid
argument in my book, since defaults exist so users could tweak them.
If it turns out a vast majority of users want savehist on by default,
then you and myself will need to adapt -- and the way to adapt is very
easy.

> Hence we should resolve this problem with savehist in advance

Sorry, I see no problem to solve here, except in the customizations of
those who routinely run multiple sessions on the same system with the
same HOME directory.

> Thus with the aforementioned disadvantages on one hand, and all the toil
> entailed on the other, is it any wonder that some people are averse to
> this proposal?

I'm open to all the opinions, I even asked to widen the audience from
which we want to hear.  But please don't expect that a small number of
objections will necessarily prevent us from changing this default, as
long as the reasons are of this nature (and not, say, some nasty
problem that we are not thinking about right now).

Re: Turning on savehist-mode by default

[Po Lu]

Eli Zaretskii <eliz@gnu.org> writes:

> Turning off savehist-mode will fix all of those, no?
>
> You are basically describing something that can happen with any Emacs
> feature that is turned on by default.  The solution is to turn them
> off when you want to avoid any of their effects.  I routinely do that
> with blink-cursor-mode, global-eldoc-mode, and sometimes others, when
> they might get in the way.  Or invoke "emacs -D".

Point taken.  Still, savehist-mode is more intrusive than those features
one might disable in the course of developing redisplay.

> That's true, but using such multiple sessions (as opposed to a single
> session) is a personal preference.  Resisting some change in the
> defaults because it adversely affects you personally is not a valid
> argument in my book, since defaults exist so users could tweak them.
> If it turns out a vast majority of users want savehist on by default,
> then you and myself will need to adapt -- and the way to adapt is very
> easy.

[...]

> Sorry, I see no problem to solve here, except in the customizations of
> those who routinely run multiple sessions on the same system with the
> same HOME directory.

The natural extension of this line of reasoning is to regard every
feature we wish to eliminate (or banish behind some user option) as a
"user preference," which is quite the slippery slope.  But that isn't
what is being proposed here, so I digress.

My reservations about this are in large part just that no evidence has
been presented demonstrating that the absence of savehist-mode or its
position as a feature disabled by default has made life difficult for
many users, who might as easily enable it in _their_ customizations.
Conjecture from practiced Emacs users (like the OP) comparing Emacs
against other like programs is of little value, and there are as many
programs that prompt users for commands to input without saving them to
a history file as there are that do.  The GIMP comes to mind; no history
is saved by any of its command dialogs, such as the one that pops up
when you type /.

Re: Turning on savehist-mode by default

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: stefankangas@gmail.com,  sbaugh@catern.com,  emacs-devel@gnu.org
> Date: Sun, 17 Dec 2023 21:31:49 +0800
> 
> My reservations about this are in large part just that no evidence has
> been presented demonstrating that the absence of savehist-mode or its
> position as a feature disabled by default has made life difficult for
> many users, who might as easily enable it in _their_ customizations.

Indeed, I think turning this on by default is only justified if a
majority of users do that anyway.

Re: Turning on savehist-mode by default

[Juergen Fenn]

Am 17.12.23 um 14:45 Uhr schrieb Eli Zaretskii:
> Indeed, I think turning this on by default is only justified if a
> majority of users do that anyway.

But how would you find out this is so?

Regards,
Jürgen.

Re: Turning on savehist-mode by default

[Eli Zaretskii]

> Date: Sun, 17 Dec 2023 18:55:18 +0100
> From: Juergen Fenn <jfenn@gmx.net>
> 
> 
> 
> Am 17.12.23 um 14:45 Uhr schrieb Eli Zaretskii:
> > Indeed, I think turning this on by default is only justified if a
> > majority of users do that anyway.
> 
> But how would you find out this is so?

That's why I suggested to poll users.

RE: [External] : Re: Turning on savehist-mode by default

[Drew Adams]

> ... Given my experience with savehist causing
> unexpected and hard-to-debug performance problems,
> I'd guess that there are more such cases in the
> wild waiting to be triggered...
> 
> As well, I have some concerns about savehist's
> having the potential to cause weird bugs in other
> libraries: The savehist-save function seems to
> comment out individual elements of
> savehist-minibuffer-history-variables
> that it determines are unreadable.  That's
> understandable from its perspective, but what
> effect will that have on libraries that may not be
> expecting for their data structures to have
> certain parts disappear after restarting Emacs?...

+1.

At the very least, make users aware that
their input history is being saved by
default.  And explicitly point them to
how to configure savehist to save only
some histories and only some additional
variables.

IOW, let's not start silently saving
stuff users type/enter, without making
them very aware this is being done.

I also don't think changing the default
behavior is _necessary_, regardless of
whether some users might expect it
because they're used to something they
think is similar to that outside Emacs.

Do user choices from menus etc. get
recorded willy nilly outside Emacs?
Emacs completion choices are in some
ways like menu choices - especially
with fancy "completion frameworks".

Lots of stuff's input in the minibuffer
nowadays (esp. with completion) - a far
cry from just command, file, and buffer
names, which was the case for the first
several Emacs decades.

Emacs end users aren't necessarily
programmers.  And I think something like
a CLI `.history' file is a bit different
from all minibuffer input history.  The
latter is a wide net with tiny holes.
___

The first step is having users _aware_
of savehist.

And I think that's _sufficient_.  It's
_trivially easy_ for users to turn ON
saving of input history.  But they need
to know that that's the case.  And if
you change the default behavior it's
even more important for users to know
(a) that's being done and (b) how to
turn it off.  (And a NEWS item isn't
sufficient to provide such awareness,
IMO.)

If the argument is that saving should be
on by default _because_ users won't know
about being able to turn it ON, then an
even more important argument, _for the
same reason of user ignorance_, applies
to not saving by default.

All of this has already been said, but
Stefan K "didn't see any serious
objections".

Which is more serious: users not having
input saved because they're unaware of
that possibility, or users having input
recorded without being aware of that or
of how to control it?

(I'd make the same argument for bookmarks
and whatever other automatic saving is
done by default, but autosaving bookmarks
has been the default forever.  And part
of the initial idea of bookmarks is
persistence.)

Dunno how important this is, but I think
it's serious enough to at least consider.

IMO, it's fine to save stuff by default
_IF_ users are aware that's happening.
But they're not, in general, so I'd say
leave savehist OFF by default.

Re: Turning on savehist-mode by default

[Juergen Fenn]

Am 17.12.23 um 19:09 Uhr schrieb Eli Zaretskii:
>> Date: Sun, 17 Dec 2023 18:55:18 +0100
>> From: Juergen Fenn <jfenn@gmx.net>
>>
>>
>>
>> Am 17.12.23 um 14:45 Uhr schrieb Eli Zaretskii:
>>> Indeed, I think turning this on by default is only justified if a
>>> majority of users do that anyway.
>>
>> But how would you find out this is so?
>
> That's why I suggested to poll users.

Right. But then, again, I wonder whether such a move would make any
sense at all. Those who have switched savehist mode on would not mind
anyway. And those who haven't already and don't want to use it, like me,
would at least be irritated about the strange behaviour.

Changing defaults with no need to do so is no good.

Please leave it as is.

Thanks and
Best regards,
Jürgen.

Re: Turning on savehist-mode by default

[Eli Zaretskii]

> Date: Sun, 17 Dec 2023 20:51:07 +0100
> From: Juergen Fenn <jfenn@gmx.net>
> 
> 
> 
> Am 17.12.23 um 19:09 Uhr schrieb Eli Zaretskii:
> >> Date: Sun, 17 Dec 2023 18:55:18 +0100
> >> From: Juergen Fenn <jfenn@gmx.net>
> >>
> >>
> >>
> >> Am 17.12.23 um 14:45 Uhr schrieb Eli Zaretskii:
> >>> Indeed, I think turning this on by default is only justified if a
> >>> majority of users do that anyway.
> >>
> >> But how would you find out this is so?
> >
> > That's why I suggested to poll users.
> 
> Right. But then, again, I wonder whether such a move would make any
> sense at all.

It does make sense to me.

> Please leave it as is.

Noted, thanks.

Re: Turning on savehist-mode by default

[Dmitry Gutov]

On 17/12/2023 21:51, Juergen Fenn wrote:
> Right. But then, again, I wonder whether such a move would make any
> sense at all. Those who have switched savehist mode on would not mind
> anyway. And those who haven't already and don't want to use it, like me,
> would at least be irritated about the strange behaviour.

That implies that all Emacs users are aware of savehist-mode, and that 
it won't ever get any new users.

Pretty pessimistic.

Re: Turning on savehist-mode by default

[Juergen Fenn]

Am 17.12.23 um 21:21 Uhr schrieb Dmitry Gutov:
> On 17/12/2023 21:51, Juergen Fenn wrote:
>> Right. But then, again, I wonder whether such a move would make any
>> sense at all. Those who have switched savehist mode on would not mind
>> anyway. And those who haven't already and don't want to use it, like me,
>> would at least be irritated about the strange behaviour.
>
> That implies that all Emacs users are aware of savehist-mode, and that
> it won't ever get any new users.
>
> Pretty pessimistic.

Well, frankly speaking, I've been using Emacs for some twenty years,
mostly for LaTeX, and I've read everything I could in order to switch to
version 29. But I _never_ heard of savehist mode before I read this thread.

YMMV, of course. Just my 2ct.

Regards,
Jürgen.

Re: Turning on savehist-mode by default

[Dmitry Gutov]

On 17/12/2023 22:38, Juergen Fenn wrote:
> 
> Am 17.12.23 um 21:21 Uhr schrieb Dmitry Gutov:
>> On 17/12/2023 21:51, Juergen Fenn wrote:
>>> Right. But then, again, I wonder whether such a move would make any
>>> sense at all. Those who have switched savehist mode on would not mind
>>> anyway. And those who haven't already and don't want to use it, like me,
>>> would at least be irritated about the strange behaviour.
>> That implies that all Emacs users are aware of savehist-mode, and that
>> it won't ever get any new users.
>>
>> Pretty pessimistic.
> Well, frankly speaking, I've been using Emacs for some twenty years,
> mostly for LaTeX, and I've read everything I could in order to switch to
> version 29. But I_never_  heard of savehist mode before I read this thread.

Then it must follow that there are a lot of existing Emacs users who 
haven't heard of savehist-mode, and who might like it (or not).

But switching it on by default can help with discovering it.

RE: [External] : Re: Turning on savehist-mode by default

[Drew Adams]

> switching it on by default can help with discovering it.

If true (and it likely is), then it's true
of _everything_ that is off by default.

Not a good argument for turning _anything_
ON by default.  Not by itself, anyway.

Re: [External] : Re: Turning on savehist-mode by default

[Dmitry Gutov]

On 17/12/2023 23:12, Drew Adams wrote:
>> switching it on by default can help with discovering it.
> If true (and it likely is), then it's true
> of_everything_  that is off by default.

Yes, it is.

But it's a counter-argument to the principle of the least inconvenience, 
presented by Juergen. Not an absolute principle.

Re: [External] : Re: Turning on savehist-mode by default

[Juergen Fenn]

Am 17.12.23 um 22:16 Uhr schrieb Dmitry Gutov:
> On 17/12/2023 23:12, Drew Adams wrote:
>>> switching it on by default can help with discovering it.
>> If true (and it likely is), then it's true
>> of_everything_  that is off by default.
>
> Yes, it is.
>
> But it's a counter-argument to the principle of the least inconvenience,
> presented by Juergen. Not an absolute principle.

Right. Generally, I agree to Drew's point of view. But on the other
hand, what would be an advantage of switching on savehist mode by
default? Maybe it would make it easier for users of other editors to
switch to Emacs because savehist mode provides a feature they are used
to and they could miss. This might be a good point indeed.

Regards,
Jürgen.

RE: [External] : Re: Turning on savehist-mode by default

[Drew Adams]

> >> switching it on by default can help with discovering it.
> >
> > If true (and it likely is), then it's true
> > of_everything_  that is off by default.
> 
> Yes, it is.
> 
> But it's a counter-argument to the principle
> of the least inconvenience,

I'm guessing you mean to say that turning
everything ON by default provides the
least inconvenience?

That begs the question of just what that
"principle" might be.

Or did you perhaps mean the principle of
least _astonishment_ (aka surprise)?

  "a component of a system should behave
   in a way that most users will expect
   it to behave, and therefore not astonish
   or surprise users"

https://en.wikipedia.org/wiki/Principle_of_least_astonishment

Are we talking about (existing) Emacs users?
Or (potential) new Emacs users?  Or both?

Do most potential new users have experience
with something like minibuffer input (esp.
with completion)?

Would most users of other apps expect that
their menu choices will be saved persistently?  (Minibuffer input with completion is akin to
choosing from a menu.)  Would saving their
menu choices astonish them less, or more,
than not saving them?

  "The choice of "least surprising" behavior
   can depend on the expected audience (for
   example, end users, programmers, or system
   administrators)." -- Wikipedia, op. cit.

Emacs users, and potential new users, can
be of any sort.  As an "expected audience"
they're all over the map.

As Eli said, "turning this on by default is
only justified if a majority of users do
that anyway."

I think Eli was talking about existing users.
But in terms of expectations it could also be
applied to potential new users: would most of
them be least surprised by saving or not
saving minibuffer inputs?

Re: [External] : Re: Turning on savehist-mode by default

[Dmitry Gutov]

On 17/12/2023 23:55, Drew Adams wrote:
>>>> switching it on by default can help with discovering it.
>>> If true (and it likely is), then it's true
>>> of_everything_  that is off by default.
>> Yes, it is.
>>
>> But it's a counter-argument to the principle
>> of the least inconvenience,
> I'm guessing you mean to say that turning
> everything ON by default provides the
> least inconvenience?

Not at all.

RE: [External] : Re: Turning on savehist-mode by default

[Drew Adams]

> >>> switching it on by default can help with discovering it.
> >>
> >> If true (and it likely is), then it's true
> >> of_everything_  that is off by default.
> >
> > Yes, it is.
> >
> > But it's a counter-argument to the principle of the least inconvenience,
> > presented by Juergen. Not an absolute principle.
> 
> Right. Generally, I agree to Drew's point of view. But on the other
> hand, what would be an advantage of switching on savehist mode by
> default? Maybe it would make it easier for users of other editors to
> switch to Emacs because savehist mode provides a feature they are used
> to and they could miss. This might be a good point indeed.

Are users of other editors really accustomed
to persistent recording of their inputs?
We're talking here about minibuffer input.

With completion - especially with some "fancy
completion frameworks", minibuffer input can
include what corresponds to menu choices in
other apps.  Do users of other editors expect
their _menu_ choices to be recorded
persistently?  I doubt it.
___

But if we think it's important to record such
choices when entered in the minibuffer, then
why not propose to have savehist record Emacs
menu choices more generally?

When you use menubar menu item File > Open
File' to open file XYZ, should savehist
record that in some `menubar-file-history'
variable?  Maybe laughable. ;-)
___

But FWIW, in Icicles there's user option
`icicle-menu-items-to-history-flag':

  `icicle-menu-items-to-history-flag' is a
  variable defined in `icicles-opt.el'.

  Its value is t

  Non-nil means to add menu-item commands to
  the command history.
  This history is `extended-command-history'.

And note that it's ON by default!  (Of course,
Icicle mode itself isn't ON by default.)

`M-x' input contributes to that command
history. And there are features (commands
etc.) that let you reuse/invoke commands in
that history - `M-x' is one such feature!
So why not also let them reuse commands
you've invoked interactively in other ways
than just `M-x'?

RE: [External] : Re: Turning on savehist-mode by default

[Drew Adams]

> >>>> switching it on by default can help with discovering it.
> >>>
> >>> If true (and it likely is), then it's true
> >>> of_everything_  that is off by default.
> >>
> >> Yes, it is.
> >>
> >> But it's a counter-argument to the principle
> >> of the least inconvenience,
> >
> > I'm guessing you mean to say that turning
> > everything ON by default provides the
> > least inconvenience?
> 
> Not at all.

Then please say what you did mean.

At the outset, "it's" == "switching it on by
default can help with discovering it".

Then, in my "it's true of _everything_ that
is off by default", "it's" == same thing:
ON by default can help with discovery.

But in your "it's a counter-argument to the
principle of the least inconvenience", just
what is "it's"?

If you mean the same thing by your "it's"
as was meant in the previous posts, then I
can only guess that you're saying that
"switching it on by default can help with
discovering it" is "a counter-argument to
the principle of least inconvenience".

Is that really what you meant?

I'd think that helping discovery is an
_argument_, not a counter-argument, for
least inconvenience.  (But again, what is
that principle?)

Re: Turning on savehist-mode by default

[Po Lu]

Dmitry Gutov <dmitry@gutov.dev> writes:

> That implies that all Emacs users are aware of savehist-mode, and that
> it won't ever get any new users.
>
> Pretty pessimistic.

What's pessimistic is the depressing vision of a world in which Emacs
users who want functionality cannot research which Emacs features might
provide it, compelling us to enable everything under the sun that they
might want, however few of them do.

Re: Turning on savehist-mode by default

[Eli Zaretskii]

> From: Po Lu <luangruo@yahoo.com>
> Cc: Juergen Fenn <jfenn@gmx.net>,  emacs-devel@gnu.org
> Date: Mon, 18 Dec 2023 08:47:57 +0800
> 
> Dmitry Gutov <dmitry@gutov.dev> writes:
> 
> > That implies that all Emacs users are aware of savehist-mode, and that
> > it won't ever get any new users.
> >
> > Pretty pessimistic.
> 
> What's pessimistic is the depressing vision of a world in which Emacs
> users who want functionality cannot research which Emacs features might
> provide it, compelling us to enable everything under the sun that they
> might want, however few of them do.

The last few messages in this thread show all the alarming indications
of a discussion that is about to become another futile bikeshedding,
whereby each participant wants to say the last word, and which then
escalates into unkind discussion where people's feelings get hurt.

So I urge people to please stop this needless go-around before it's
too late.  I assure you that nothing you can say in this manner can
possibly affect the decision, whatever it will be.

Re: Turning on savehist-mode by default

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > So I see nothing special in the incident you referred to: it's what I
  > see every day.

I think we should all be more hesitate to reach a harsh conclusion
based on seeing some sort of problem.  Every program has bugs.
When people report bugs, our mission is to debug them and fix them.

But whether a program has a bigger or deeper implementation problem
that is something we can't tell, except from broader experience and
knowledge about it.


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Turning on savehist-mode by default

[Stefan Kangas]

Adam Porter <adam@alphapapa.net> writes:

> It doesn't seem like an easy problem.  My guess as to the most "correct"
> solution would be to limit how much time can be spent serializing the
> data, both as a whole and individual elements, and to cancel or
> interrupt the serialization of the whole or a part if the limit is
> exceeded--the goal being to avoid UI pauses.  Then the limit could be
> disabled when desired (e.g. when exiting Emacs, wanting to be sure the
> data is saved).  But doing that would require some bookkeeping in the
> background, which seems like it might be excessive complexity for what
> should be a simple feature.

Thinking about this a bit more, why is this not to be considered a bug
in ement?

It does seem like that package is quite unusual in that it stores both
non-serializable and large data in the -history variable.  Perhaps we
should simply say that you should not do that?

Re: Turning on savehist-mode by default

[Visuwesh]

[வெள்ளி டிசம்பர் 22, 2023] Stefan Kangas wrote:

> Adam Porter <adam@alphapapa.net> writes:
>
>> It doesn't seem like an easy problem.  My guess as to the most "correct"
>> solution would be to limit how much time can be spent serializing the
>> data, both as a whole and individual elements, and to cancel or
>> interrupt the serialization of the whole or a part if the limit is
>> exceeded--the goal being to avoid UI pauses.  Then the limit could be
>> disabled when desired (e.g. when exiting Emacs, wanting to be sure the
>> data is saved).  But doing that would require some bookkeeping in the
>> background, which seems like it might be excessive complexity for what
>> should be a simple feature.
>
> Thinking about this a bit more, why is this not to be considered a bug
> in ement?
>
> It does seem like that package is quite unusual in that it stores both
> non-serializable and large data in the -history variable.  Perhaps we
> should simply say that you should not do that?

AFAIU the problem, ement does not store any such data in -history
variable.  It passes those large data as arguments to oft used functions
and these calls get added to command-history.  Over time the history
grows large because serialisation time for these ement forms start to
accumulate.

Re: Turning on savehist-mode by default

[Adam Porter]

On 12/22/23 04:45, Stefan Kangas wrote:
> Adam Porter <adam@alphapapa.net> writes:
> 
>> It doesn't seem like an easy problem.  My guess as to the most "correct"
>> solution would be to limit how much time can be spent serializing the
>> data, both as a whole and individual elements, and to cancel or
>> interrupt the serialization of the whole or a part if the limit is
>> exceeded--the goal being to avoid UI pauses.  Then the limit could be
>> disabled when desired (e.g. when exiting Emacs, wanting to be sure the
>> data is saved).  But doing that would require some bookkeeping in the
>> background, which seems like it might be excessive complexity for what
>> should be a simple feature.
> 
> Thinking about this a bit more, why is this not to be considered a bug
> in ement?
> 
> It does seem like that package is quite unusual in that it stores both
> non-serializable and large data in the -history variable.  Perhaps we
> should simply say that you should not do that?

There seems to be a misunderstanding.  No code in Ement stores any data 
in the command-history variable.  As I linked earlier, please see 
<https://github.com/alphapapa/ement.el/issues/216>.  What happens is 
that the arguments to interactive commands are automatically saved in 
the command-history variable by Emacs.

In Ement's case, those interactive arguments can include session and 
room structs, which store events received in the user's rooms.  Over 
time, as a session remains connected for a while (e.g. days or weeks), 
or if the user is in many, busy rooms, the amount of data can grow large 
(into the hundreds of megabytes when serialized).

When a user enables savehist-mode, it automatically serializes the 
command-history variable, which includes not only the list of commands, 
but, as I mentioned, the arguments passed to them.  As it does so, it 
writes the objects to a buffer (which, in this case, amounted to over 
400 MB for one user), then tries to read the data back to determine 
whether it's readable (which I assume would allocate the equivalent 
amount of memory again), and then, if it is readable, it serializes it 
again as it writes it to disk.

By default, savehist writes this data every 5 minutes.  So, for some of 
these users, every 5 minutes savehist was serializing 800 MB and reading 
400 MB (and writing 400 MB to disk)--as well as whenever Emacs exited.

This was bewildering until the cause was finally noticed, after months 
of these weird reports, when someone finally took the time to leave the 
profiler running and look at the report.  Of course, it only happened to 
the few users who activated savehist-mode, and none of them attributed 
the problem to savehist, which doesn't appear to have any relation to 
Ement whatsoever.

So I had to implement a special workaround in Ement to remove any Ement 
commands from the command-history variable before savehist can try to 
save it.

I hope it's now clear what I've been trying to explain, and why I've 
done so.  If there are or come to be any other packages out there that 
do something similar with command arguments, and savehist is suddenly 
enabled when a user upgrades to a new version of Emacs, it's unlikely 
that users of such packages will attribute this weird behavior to 
savehist.  And as the command-history variable, and perhaps others, grow 
over time, as savehist continues to save and restore them, performance 
may gradually decline.  As well, this can cause unnecessary, excessive 
disk writes, which isn't good for SSDs.

But if these issues do not seem likely to be problematic, then I hope 
that is so.  I have no more to say on this topic.

Re: Turning on savehist-mode by default

[Yuri Khan]

On Fri, 22 Dec 2023 at 20:19, Adam Porter <adam@alphapapa.net> wrote:


> There seems to be a misunderstanding.  No code in Ement stores any data
> in the command-history variable.  As I linked earlier, please see
> <https://github.com/alphapapa/ement.el/issues/216>.  What happens is
> that the arguments to interactive commands are automatically saved in
> the command-history variable by Emacs.
>
> In Ement's case, those interactive arguments can include session and
> room structs, which store events received in the user's rooms.  Over
> time, as a session remains connected for a while (e.g. days or weeks),
> or if the user is in many, busy rooms, the amount of data can grow large
> (into the hundreds of megabytes when serialized).

Maybe the problem could also be solved by storing session and room
structs in a hash, keyed with something that could be used in a
subsequent session, and commands accepting keys rather than full
structs?