isolating history with buffer-local variables

isolating history with buffer-local variables

[Nick Helm]

Hello, I'm trying to teach myself some elisp, but I'm having a problem with some code that works with buffer-local variables. It seems to work as intended for some variables, but not others.

My objective is to create a minor mode that makes Emacs less likely to leak private data. 

The approach I've taken is to locally turn off backups and auto-saves, and create buffer-local variables to isolate command histories and rings that may accumulate private information. My intent is for editing to work normally in the private buffer, with local histories and the local kill-ring etc all available, but without exposing them globally to other buffers and packages like savehist, which should continue uninterrupted. When the minor mode is turned off, the local variables are killed, reinstating the global bindings.

Here's the code:

(define-minor-mode private-mode
   "This mode prevents Emacs from creating backups, auto-saves and
   certain history records for the current buffer. It is useful
   when editing private data, such as password files, key-chains and
   secure notes."
   :init-value nil
   :lighter " Private"
   :keymap nil
   (if (symbol-value private-mode) (progn
      (setq private-variables '( ;; data to isolate
         minibuffer-history         ;; <-- not working
         command-history            ;; ok
         extended-command-history   ;; <-- not working
         string-rectangle-history   ;; <-- not working
         query-replace-history      ;; ok
         search-ring                ;; ok
         regexp-search-ring         ;; ok
         kill-ring                  ;; ok
         backup-inhibited           ;; ok
         auto-save-timeout))        ;; ok 
      (dolist (variable private-variables) (make-local-variable variable)) ;; make local vars
      (setq backup-inhibited t) ;; locally disable backups
      (setq auto-save-timeout 0)) ;; locally idle auto-saves
      ;; TODO: Add idle timer to purge changes to local vars
    (dolist (variable private-variables) (kill-local-variable variable)))) ;; pop local vars

All the local auto-save/backup vars, local ring vars, and local query-replace-history and command-history vars work as intended. But minibuffer-history, extended-command-history and string-rectangle-history do not. The buffer-local vars for these are made as expected, but they are ignored and histories continue to accumulate in the global variables.

Any idea why? Anyone have suggestions for a different approach or a way around the problem? 

(There is a somewhat relevant discussion here: http://lists.gnu.org/archive/html/help-gnu-emacs/2011-10/msg00292.html)

Nick

Re: isolating history with buffer-local variables

[Stefan Monnier]

Thanks.  I think this functionality is important and should largely be
activated automatically when visiting encrypted files (not sure which
other files/buffers would warrant such behavior).

So I'm happy to see you work on this, and I encourage you try and make
it work well, and contribute your code so we can include it directly
in Emacs.

>    (if (symbol-value private-mode)

`private-mode' will normally have value either t or nil, and the
symbol-value of nil is nil and the symbol-value of t is t, so the above
is better written as "(if private-mode".

>       (setq private-variables '( ;; data to isolate
>          minibuffer-history         ;; <-- not working
>          command-history            ;; ok
>          extended-command-history   ;; <-- not working
>          string-rectangle-history   ;; <-- not working
>          query-replace-history      ;; ok
>          search-ring                ;; ok
>          regexp-search-ring         ;; ok
>          kill-ring                  ;; ok
>          backup-inhibited           ;; ok
>          auto-save-timeout))        ;; ok 

This should probably be moved out to a defvar.

>       (setq backup-inhibited t) ;; locally disable backups
>       (setq auto-save-timeout 0)) ;; locally idle auto-saves

For encrypted files (accessed via EPA), the above should be correctly
handled already (either by preventing autosave/backups or by keeping
those files encrypted just like the main file).  If you find they're
not, please file it as a bug.

> All the local auto-save/backup vars, local ring vars, and local
> query-replace-history and command-history vars work as intended. But
> minibuffer-history, extended-command-history and
> string-rectangle-history do not. The buffer-local vars for these are
> made as expected, but they are ignored and histories continue to
> accumulate in the global variables.

I think Pascal has the right explanation for that.

> Any idea why? Anyone have suggestions for a different approach or
> a way around the problem?

You'll probably need to hack read-from-minibuffer (e.g. with an advice)
so as to redirect the history variable to another variable, or to
cleanup the variable after the fact.


        Stefan

Re: isolating history with buffer-local variables

[Nick Helm]

> On 14/05/2015, at 9:39 am, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
> Thanks.  I think this functionality is important and should largely be
> activated automatically when visiting encrypted files (not sure which
> other files/buffers would warrant such behavior).

I could add to auto-mode-alist to activate private-mode for .gpg files 
(for example), but I don't know if the file extension is a reliable 
enough way to identify an encrypted file. I'll have a look at epa-hook
instead. 

I think there are other use-cases though. For instance, I use Emacs to 
edit confidential files that I don't want leaking into incremental 
backups or off-site syncing of ~/.emacs.d/ (all the usual financial,
medical, legal stuff).

> So I'm happy to see you work on this, and I encourage you try and make
> it work well, and contribute your code so we can include it directly
> in Emacs.

Gladly. I'll progress it as far as I can then ask for further critique
and suggestions.

>>   (if (symbol-value private-mode)
> 
> `private-mode' will normally have value either t or nil, and the
> symbol-value of nil is nil and the symbol-value of t is t, so the above
> is better written as "(if private-mode".
> 
>>      (setq private-variables '( ;; data to isolate
>>         minibuffer-history         ;; <-- not working
>>         command-history            ;; ok
>>         extended-command-history   ;; <-- not working
>>         string-rectangle-history   ;; <-- not working
>>         query-replace-history      ;; ok
>>         search-ring                ;; ok
>>         regexp-search-ring         ;; ok
>>         kill-ring                  ;; ok
>>         backup-inhibited           ;; ok
>>         auto-save-timeout))        ;; ok 
> 
> This should probably be moved out to a defvar.
> 
>>      (setq backup-inhibited t) ;; locally disable backups
>>      (setq auto-save-timeout 0)) ;; locally idle auto-saves
> 
> For encrypted files (accessed via EPA), the above should be correctly
> handled already (either by preventing autosave/backups or by keeping
> those files encrypted just like the main file).  If you find they're
> not, please file it as a bug.

Yes, that's what I see here too.

>> All the local auto-save/backup vars, local ring vars, and local
>> query-replace-history and command-history vars work as intended. But
>> minibuffer-history, extended-command-history and
>> string-rectangle-history do not. The buffer-local vars for these are
>> made as expected, but they are ignored and histories continue to
>> accumulate in the global variables.
> 
> I think Pascal has the right explanation for that.

I thought that was probably the case, but I couldn't pin down why some 
minibuffer input was prepended to the local variable 
(e.g. command-history) while other input was not 
(e.g. extended-command-history).

>> Any idea why? Anyone have suggestions for a different approach or
>> a way around the problem?
> 
> You'll probably need to hack read-from-minibuffer (e.g. with an advice)
> so as to redirect the history variable to another variable, or to
> cleanup the variable after the fact.

Thanks, I'll give those a try.

> 
>        Stefan
> 
> 
> 

Re: isolating history with buffer-local variables

[Stefan Monnier]

> I think there are other use-cases though. For instance, I use Emacs to
> edit confidential files that I don't want leaking into incremental
> backups or off-site syncing of ~/.emacs.d/ (all the usual financial,
> medical, legal stuff).

Could you give details of those use-cases, to see how much of it is very
specific to your particular setup, and how much of it could apply to
a wider audience?

I'm specifically wondering why copying "foo" to "foo~" or to "#foo#"
would be a problem (assuming "foo" itself is modified) w.r.t
incremental backups or off-site syncing.  Or are your worries more
specifically about leaking data from those files via history variables
saved in ~/.emacs.d ?
[ Another way to attack this problem is to try and avoid
  syncing/backing-up the saved-history data in ~/.emacs.d.
  Maybe this history data should never be saved to ~/.emacs.d but to
  another place instead?  ]

>>> (setq backup-inhibited t) ;; locally disable backups
>>> (setq auto-save-timeout 0)) ;; locally idle auto-saves
>> For encrypted files (accessed via EPA), the above should be correctly
>> handled already (either by preventing autosave/backups or by keeping
>> those files encrypted just like the main file).  If you find they're
>> not, please file it as a bug.
> Yes, that's what I see here too.

In which other situations did you find such settings to be needed?

> I thought that was probably the case, but I couldn't pin down why some
> minibuffer input was prepended to the local variable
> (e.g. command-history) while other input was not
> (e.g. extended-command-history).

`command-history' is a variable managed by the read-eval command loop,
whereas `extended-command-history' is a normal minibuffer history
variable (i.e. managed by read-from-minibuffer).

Minibuffer history variable are apparently set while we're in the
minibuffer.  Maybe we should change this so that minibuffer history
variables get set in the buffer from which read-from-minibuffer is
called instead of in the minibuffer.  It would probably be very useful
in your case, and seems like a good idea in general.
I suggest you file a bug report about that.


        Stefan

Re: isolating history with buffer-local variables

[Nick Helm]

> On 16/05/2015, at 11:32 am, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>> I think there are other use-cases though. For instance, I use Emacs to
>> edit confidential files that I don't want leaking into incremental
>> backups or off-site syncing of ~/.emacs.d/ (all the usual financial,
>> medical, legal stuff).
> 
> Could you give details of those use-cases, to see how much of it is very
> specific to your particular setup, and how much of it could apply to
> a wider audience?
> 
> I'm specifically wondering why copying "foo" to "foo~" or to "#foo#"
> would be a problem (assuming "foo" itself is modified) w.r.t
> incremental backups or off-site syncing.  Or are your worries more
> specifically about leaking data from those files via history variables
> saved in ~/.emacs.d ?
> [ Another way to attack this problem is to try and avoid
>  syncing/backing-up the saved-history data in ~/.emacs.d.
>  Maybe this history data should never be saved to ~/.emacs.d but to
>  another place instead?  ]

My primary concern is losing control of copies or fragments of confidential files. 

If I edit a confidential file foo.txt, which is stored in plain text on my file-server, Emacs potentially creates at least three additional copies or fragments of foo.txt created on the local machine - foo.txt~x~, #foo.txt# and .emacs-history. 

My file-server does encrypted backups to keep the originals secure, but I also I do hourly incremental rsync backups of $HOME on each client machine, so another three copies and fragments of foo.txt find their way to that machine as well. These backups are rotated off-site.

I also sync ~/.emacs.d to Dropbox for continuity and so I can access my Emacs setup, backups, auto-saves and histories from any machine, but that means another three copies and fragments of foo.txt now exist up on Dropbox.

So, all up, for a single edit, I could end up with nine or more uncontrolled copies or fragments of foo.txt floating around. For normal files, this is a good thing, but not for private data (to my mind at least). Where the edit takes place on a shared machine or a portable device, there is also an additional risk of theft.

Similarly, if I use Tramp to remote into a client's machine and edit foo.conf for admin purposes, Emacs creates foo.conf~x~ #foo.conf# and saves file fragments in .emacs-history on the local machine. The remote sysadmin can add file variables:

;; backup-inhibited: t
;; eval: (auto-save-mode 0)

to foo.conf to prevent the copies being created, but has no simple control over saved rings and histories.

Out of interest, I ran a series of little tests to see what Emacs saves in different situations. This table shows the data that was committed to the local disk when editing plain text and encrypted files (via EPA) in different locations. 

                    +-------------------------------+
                    |File content                   |
                    +---------------+---------------+
                    |plain-text     |encrypted      |
+---------+---------+---------------+---------------+
|File     |Local FS |Fp Ap Bp Hp Rp |Fe -  Be Hp Rp |
|loc      +---------+---------------+---------------+
|         |LAN      |Fp Ap Bp Hp Rp |Fe -  Be Hp Rp |
|         +---------+---------------+---------------+
|         |Remote   |Fp Ap Bp Hp Rp |Fe Ap Be Hp Rp |
+---------+---------+---------------+---------------+

Where:
F = visited file (on user save)
A = auto-saves (to ~/.emacs.d/auto-saves)
B = backups (to ~/.emacs.d/backups)
H = minibuffer history (et al) (to ~/.emacs.d/.emacs_history via savehist)
R = kill-ring (et al) (also to ~/.emacs.d/.emacs_history via savehist)

And:
p = data was saved to disk in plain text
e = data was saved to disk encrypted

The results are largely as expected, with the exception of the handling of auto-saves on remote machines (the remote encrypted gpg was saved in plain text on the local machine). Perhaps this is a bug.

That issue aside, I hope private-mode will enable something like the following. That is, make it easier to control what private data is committed to disk without an explicit user save.

               +---------------------------------------------------------------+
               |File content                                                   |
               +---------------+-------------------------------+---------------+ 
                               |            PRIVATE            |       
               +---------------+---------------+---------------+---------------+
               |plain-text     |plain-text     |encrypted      |encrypted      |
+----+---------+---------------+---------------+---------------+---------------+
|File|Local FS |Fp Ap Bp Hp Rp |Fp -  -  -  -  |Fe -  -  -  -  |Fe -  Be Hp Rp |
|loc +---------+---------------+---------------+---------------+---------------+
|    |LAN      |Fp Ap Bp Hp Rp |Fp -  -  -  -  |Fe -  -  -  -  |Fe -  Be Hp Rp |
|    +---------+---------------+---------------+---------------+---------------+
|    |Remote   |Fp Ap Bp Hp Rp |Fp -  -  -  -  |Fe -  -  -  -  |Fe -  Be Hp Rp |
+----+---------+---------------+---------------+---------------+---------------+


Nick

Re: isolating history with buffer-local variables

[Stefan Monnier]

> So, all up, for a single edit, I could end up with nine or more uncontrolled
> copies or fragments of foo.txt floating around. For normal files, this

To the extent that 9 copies is not fundamentally different from
3 copies, I think that it's OK to require the user to explicitly request
to skip backups/autosaves via something like:

> ;; backup-inhibited: t
> ;; eval: (auto-save-mode 0)

[ I have backup inhibited globally, FWIW.  ]

> to foo.conf to prevent the copies being created, but has no simple control
> over saved rings and histories.

For rings and history, the problem here is the same as for encrypted
files, so once we solve the problem for encrypted files, it can be used
for these confidential-but-not-encrypted files.

> +---------+---------+---------------+---------------+
> |File     |Local FS |Fp Ap Bp Hp Rp |Fe -  Be Hp Rp |
> |loc      +---------+---------------+---------------+
> |         |LAN      |Fp Ap Bp Hp Rp |Fe -  Be Hp Rp |
> |         +---------+---------------+---------------+
> |         |Remote   |Fp Ap Bp Hp Rp |Fe Ap Be Hp Rp |
> +---------+---------+---------------+---------------+
>
> Where:
> F = visited file (on user save)
> A = auto-saves (to ~/.emacs.d/auto-saves)
> B = backups (to ~/.emacs.d/backups)
> H = minibuffer history (et al) (to ~/.emacs.d/.emacs_history via savehist)
> R = kill-ring (et al) (also to ~/.emacs.d/.emacs_history via savehist)

As mentioned, please report a bug w.r.t minibuffer histories so your
private-mode can more easily make sure data doesn't escape this way.

> The results are largely as expected, with the exception of the handling of
> auto-saves on remote machines (the remote encrypted gpg was saved in plain
> text on the local machine). Perhaps this is a bug.

This is very much a severe bug, yes.  Please report it.

>                +---------------+-------------------------------+---------------+ 
>                                |            PRIVATE            |       
>                +---------------+---------------+---------------+---------------+
>                |plain-text     |plain-text     |encrypted      |encrypted      |
> +----+---------+---------------+---------------+---------------+---------------+

I'm not sure we need to distinguish "encrypted+private" from the "normal
encrypted" files.  I.e. we should treat all encrypted files as private
(at least by default).


        Stefan

Re: isolating history with buffer-local variables

[Stefan Monnier]

> I also sync ~/.emacs.d to Dropbox for continuity and so I can access my

BTW, this sounds rather shocking given your concern for privacy.
After all, Dropbox can change those files at will.


        Stefan

Re: isolating history with buffer-local variables

[Nick Helm]

> On 25/05/2015, at 12:39 pm, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>> I also sync ~/.emacs.d to Dropbox for continuity and so I can access my
> 
> BTW, this sounds rather shocking given your concern for privacy.
> After all, Dropbox can change those files at will.
> 

I agree, it's not ideal! But it's a wonderful (read easy) way to provide persistence between Emacs sessions on different devices in different locations.

How do other users achieve continuity between sessions?

Re: isolating history with buffer-local variables

[Pascal J. Bourguignon]

Nick Helm <nick@tenpoint.co.nz> writes:

>> On 25/05/2015, at 12:39 pm, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>> 
>>> I also sync ~/.emacs.d to Dropbox for continuity and so I can access my
>> 
>> BTW, this sounds rather shocking given your concern for privacy.
>> After all, Dropbox can change those files at will.
>> 
>
> I agree, it's not ideal! But it's a wonderful (read easy) way to
> provide persistence between Emacs sessions on different devices in
> different locations.
>
> How do other users achieve continuity between sessions?

I never quit emacs.

(emacs-uptime)
 --> "Up 83d 2h 21m 42s (Tue 2015-03-03 00:50:26), 68 buffers, 1 files"

-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk

Re: isolating history with buffer-local variables

[Nick Helm]

> On 25/05/2015, at 2:12 pm, Pascal J. Bourguignon <pjb@informatimago.com> wrote:
> 
>> How do other users achieve continuity between sessions?
> 
> I never quit emacs.
> 
> (emacs-uptime)
> --> "Up 83d 2h 21m 42s (Tue 2015-03-03 00:50:26), 68 buffers, 1 files"

How do you achieve that - do you always work on a single device, use 
emacsclient against a remote emacs server process, or something else?

Re: isolating history with buffer-local variables

[Pascal J. Bourguignon]

Nick Helm <nick@tenpoint.co.nz> writes:

> Hello, I'm trying to teach myself some elisp, but I'm having a problem
> with some code that works with buffer-local variables. It seems to
> work as intended for some variables, but not others.
>
> My objective is to create a minor mode that makes Emacs less likely to
> leak private data. 
>
> The approach I've taken is to locally turn off backups and auto-saves,
> and create buffer-local variables to isolate command histories and
> rings that may accumulate private information. My intent is for
> editing to work normally in the private buffer, with local histories
> and the local kill-ring etc all available, but without exposing them
> globally to other buffers and packages like savehist, which should
> continue uninterrupted. When the minor mode is turned off, the local
> variables are killed, reinstating the global bindings.
>
> Here's the code:
>
> (define-minor-mode private-mode
>    "This mode prevents Emacs from creating backups, auto-saves and
>    certain history records for the current buffer. It is useful
>    when editing private data, such as password files, key-chains and
>    secure notes."
>    :init-value nil
>    :lighter " Private"
>    :keymap nil
>    (if (symbol-value private-mode) (progn
>       (setq private-variables '( ;; data to isolate
>          minibuffer-history         ;; <-- not working
>          command-history            ;; ok
>          extended-command-history   ;; <-- not working
>          string-rectangle-history   ;; <-- not working
>          query-replace-history      ;; ok
>          search-ring                ;; ok
>          regexp-search-ring         ;; ok
>          kill-ring                  ;; ok
>          backup-inhibited           ;; ok
>          auto-save-timeout))        ;; ok 
>       (dolist (variable private-variables) (make-local-variable variable)) ;; make local vars
>       (setq backup-inhibited t) ;; locally disable backups
>       (setq auto-save-timeout 0)) ;; locally idle auto-saves
>       ;; TODO: Add idle timer to purge changes to local vars
>     (dolist (variable private-variables) (kill-local-variable variable)))) ;; pop local vars
>
> All the local auto-save/backup vars, local ring vars, and local
> query-replace-history and command-history vars work as intended. But
> minibuffer-history, extended-command-history and
> string-rectangle-history do not. The buffer-local vars for these are
> made as expected, but they are ignored and histories continue to
> accumulate in the global variables.
>
> Any idea why? Anyone have suggestions for a different approach or a
> way around the problem? 
>
> (There is a somewhat relevant discussion here:
> http://lists.gnu.org/archive/html/help-gnu-emacs/2011-10/msg00292.html)

minibuffer-history is used in the minibuffer.  This is a different
buffer.  Therefore it's a different buffer local variable.

I guess a similar problem might be the case for the other variables.


-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk