How to disable org-persist in a given file?

How to disable org-persist in a given file?

[Gustavo Barros]

Hi All,

this is more of a question really. I moved recently to the new pretest
version of Emacs and thus got the 9.6 version of Org. And after that I
started to notice some saving by `org-persist' going on, and indeed I
have a new associated "~/.cache/org-persist/" directory.

However, I do have some Org files which contain sensitive data. I'd
like to be able to disable persistence in those files. Now, some of
them are ".org.gpg" files, and I've seen the sources of `org-persist'
and noticed persistence is inhibited for them. But I also do have some
encrypted loop devices which, once opened, have a "plain" .org file
there.

I've greped the sources, and I see that `org-element' is making use of
`org-persist'. I've also seen the release notes, and that
`org-element' makes available `org-element-cache-persistent'.

However, taking a look at `org-persist's API, I couldn't find any
obvious handle to disable persistence there. Everything I've seen
presumes I have knowledge of registered "containers", which I don't.

Is currently `org-element' the only part of Org using persistence by
default? Is there some way to inhibit `org-persist' altogether in a
given file? (As opposed to just inhibiting `org-element' use of it
with `org-element-cache-persistent').

Best regards,
Gustavo.

Re: How to disable org-persist in a given file?

[Ihor Radchenko]

Gustavo Barros <gusbrs.2016@gmail.com> writes:

> However, I do have some Org files which contain sensitive data. I'd
> like to be able to disable persistence in those files. Now, some of
> them are ".org.gpg" files, and I've seen the sources of `org-persist'
> and noticed persistence is inhibited for them. But I also do have some
> encrypted loop devices which, once opened, have a "plain" .org file
> there.

Thanks for letting us know about this scenario!

> However, taking a look at `org-persist's API, I couldn't find any
> obvious handle to disable persistence there. Everything I've seen
> presumes I have knowledge of registered "containers", which I don't.
>
> Is currently `org-element' the only part of Org using persistence by
> default? Is there some way to inhibit `org-persist' altogether in a
> given file? (As opposed to just inhibiting `org-element' use of it
> with `org-element-cache-persistent').

You can use `org-persist-before-write-hook' to disable writing
selectively.

You can refer to the comment in org-persist.el for explanation about the
core concepts about CONTAINER and ASSOCIATED terms.

Let us know if you have difficulties understanding the commentary or if
you think that things can be improved.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: How to disable org-persist in a given file?

[Gustavo Barros]

Hi Ihor,

On Sun, 23 Apr 2023 at 07:55, Ihor Radchenko <yantar92@posteo.net> wrote:

> Thanks for letting us know about this scenario!

Yes, but there's little Emacs/Org can do there, I think. Once the
volume is mounted, there's no way to tell it is meant to be treated as
an encrypted file.

With time I'm learning that non-standard setups, while they may seem
attractive at first, come and bite you sooner or later. Not
necessarily because they are technically flawed, but because it breaks
people's expectations.

I do have some unusual partitioning scheme, and sometimes it becomes
an uphill battle to avoid leakages. On Emacs alone, I have to take
special care of backups, bookmarks (that's now fixed on 29), trash,
and now cache.

> You can use `org-persist-before-write-hook' to disable writing
> selectively.

Thanks! That's the one. Though it would be nice if a variable existed
for the purpose. `(add-hook 'org-persist-before-write-hook (lambda
(&rest _args) t) nil t)' is not our average file local variable. :)

> You can refer to the comment in org-persist.el for explanation about the
> core concepts about CONTAINER and ASSOCIATED terms.
>
> Let us know if you have difficulties understanding the commentary or if
> you think that things can be improved.

I took a closer look at it now, and I think it is clear. But I do have
a suggestion. I've seen the use of `org-persist' by `org-element' by
default announced in the news file. But I think this would deserve an
entry in the manual (as far as I can tell, currently there isn't one,
but I'm running built-in on 29, so I might be out of date), letting
people know it is enabled by default, and how to opt out, if they want
to. As my scenario above shows, there's little hope of being able to
cover "all cases", and people must take care of that for themselves
within reason. All Org can do is let people know, and on security
related issues, better be outspoken than shy.

And thank you very much for this nice feature!

Best,
Gustavo.

Re: How to disable org-persist in a given file?

[Ruijie Yu via General discussions about Org-mode.]

Gustavo Barros <gusbrs.2016@gmail.com> writes:
>> You can use `org-persist-before-write-hook' to disable writing
>> selectively.
>
> Thanks! That's the one. Though it would be nice if a variable existed
> for the purpose. `(add-hook 'org-persist-before-write-hook (lambda
> (&rest _args) t) nil t)' is not our average file local variable. :)

    (lambda (&rest _args) t)

Side track: remember that you have `always' and `ignore' in your
collection of functions. :)

-- 
Best,


RY

[Please note that this mail might go to spam due to some
misconfiguration in my mail server -- still investigating.]

Re: How to disable org-persist in a given file?

[Ihor Radchenko]

Gustavo Barros <gusbrs.2016@gmail.com> writes:

> On Sun, 23 Apr 2023 at 07:55, Ihor Radchenko <yantar92@posteo.net> wrote:
>
>> Thanks for letting us know about this scenario!
>
> Yes, but there's little Emacs/Org can do there, I think. Once the
> volume is mounted, there's no way to tell it is meant to be treated as
> an encrypted file.

We can try (string-match mounted-file-systems default-directory).
Will it work with your setup?

>> You can use `org-persist-before-write-hook' to disable writing
>> selectively.
>
> Thanks! That's the one. Though it would be nice if a variable existed
> for the purpose. `(add-hook 'org-persist-before-write-hook (lambda
> (&rest _args) t) nil t)' is not our average file local variable. :)

We cannot just disable persistence completely.
For example, remote image export relies on persistence to be working
_during_ Emacs session.

What about file/directory-local variable that will redirect where to
save cache? 

> ... I think this would deserve an
> entry in the manual (as far as I can tell, currently there isn't one,
> but I'm running built-in on 29, so I might be out of date), letting
> people know it is enabled by default, and how to opt out, if they want
> to. As my scenario above shows, there's little hope of being able to
> cover "all cases", and people must take care of that for themselves
> within reason. All Org can do is let people know, and on security
> related issues, better be outspoken than shy.

I think we can add a section near "Code Evaluation and Security Issues".

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: How to disable org-persist in a given file?

[Gustavo Barros]

On Sun, 23 Apr 2023 at 10:51, Ihor Radchenko <yantar92@posteo.net> wrote:

> We can try (string-match mounted-file-systems default-directory).
> Will it work with your setup?

Wouldn't that exclude a lot of legitimate use cases?
Personally, I don't see an issue in this scenario of mine for Org to
handle. It's an exception I must handle.

> We cannot just disable persistence completely.
> For example, remote image export relies on persistence to be working
> _during_ Emacs session.
>
> What about file/directory-local variable that will redirect where to
> save cache?

I don't understand that. Isn't persistence about keeping data across
Emacs sessions?
Redirecting the cache is useful too. But, imho, there should be a way
to disable persistence altogether for sensitive data (or, at least,
any storage of data in other files, since I don't get what you mean by
persistence during a session). If `org-persist-before-write-hook' does
this, as I understand it does, adding a function there which just
returns the value of a variable defaulting to nil and safe-local if
booleanp, would be enough, wouldn't it? Or what am I missing?

> I think we can add a section near "Code Evaluation and Security Issues".

That would be appreciated, thank you.

Best,
Gustavo.

Re: How to disable org-persist in a given file?

[Gustavo Barros]

On Sun, 23 Apr 2023 at 11:19, Ruijie Yu <ruijie@netyu.xyz> wrote:

> Side track: remember that you have `always' and `ignore' in your
> collection of functions. :)

Somehow I wasn't aware of `always'. Thanks!

Gustavo.

Re: How to disable org-persist in a given file?

[Ihor Radchenko]

Gustavo Barros <gusbrs.2016@gmail.com> writes:

>> We cannot just disable persistence completely.
>> For example, remote image export relies on persistence to be working
>> _during_ Emacs session.
>
> I don't understand that. Isn't persistence about keeping data across
> Emacs sessions?

Not only. We also use org-persist to cache downloaded images from
internet and, in future, to cache image previews (currently, they just
sit in `org-preview-latex-image-directory'). These scenarios make use of
org-persist during a single Emacs session, not just across several
sessions.

Even if we allow completely disabling persistence for certain files,
previews, and internet downloads should happen _somewhere_ in file system
and might inevitably cause leakage. But what is the way around?

CCing Timothy as he might have further thoughts.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: How to disable org-persist in a given file?

[Gustavo Barros]

On Tue, 25 Apr 2023 at 07:21, Ihor Radchenko <yantar92@posteo.net> wrote:

> > I don't understand that. Isn't persistence about keeping data across
> > Emacs sessions?
>
> Not only. We also use org-persist to cache downloaded images from
> internet and, in future, to cache image previews (currently, they just
> sit in `org-preview-latex-image-directory'). These scenarios make use of
> org-persist during a single Emacs session, not just across several
> sessions.

I see. But, for uses where the data is only required during a session,
does the cache have to be written to an external file? But I think I
get why write it for the previews, and obviously, for downloads too.
I'm not sure this should be called "persistence" though.

> Even if we allow completely disabling persistence for certain files,
> previews, and internet downloads should happen _somewhere_ in file system
> and might inevitably cause leakage. But what is the way around?

My view is that there really should be a way of doing it. Because this
has relevant security implications. And because users should be able
to have control of their data in the first place. And then if one such
a feature, necessarily requiring persistence (as opposed to the
convenience of "just" speeding things up), a user-error could be
signaled saying "sorry, can't do that because you disabled
persistence". And the two cases you mentioned seem quite the exception
to me. Of course, I'm speaking here about writing information about /
of the org file itself to an external file, naturally internet
downloads must go somewhere and that should be easy. The thing is Org
is used for a lot of things. You can rest assured I have no intention
of exporting my encrypted files alongside some remote content. But I
do wish to be able to make sure my passwords are safe.

Re: How to disable org-persist in a given file?

[Gustavo Barros]

On Tue, 25 Apr 2023 at 07:52, Gustavo Barros <gusbrs.2016@gmail.com> wrote:

> My view is that there really should be a way of doing it. Because this
> has relevant security implications. And because users should be able
> to have control of their data in the first place. And then if one such
> a feature, necessarily requiring persistence (as opposed to the
> convenience of "just" speeding things up), a user-error could be
> signaled saying "sorry, can't do that because you disabled
> persistence". And the two cases you mentioned seem quite the exception
> to me. Of course, I'm speaking here about writing information about /
> of the org file itself to an external file, naturally internet
> downloads must go somewhere and that should be easy. The thing is Org
> is used for a lot of things. You can rest assured I have no intention
> of exporting my encrypted files alongside some remote content. But I
> do wish to be able to make sure my passwords are safe.

For the record, even ".org.gpg" files generate an entry in the cache
index. (True, not the `:persist-file' itself though).

My ~/.cache/org-persist/index contains:

    (:container
     ((elisp org-element--headline-cache)
      (elisp org-element--cache))
     :persist-file "c8/fd2b62-45cc-41c8-8571-d944c76b1f15" :associated
     (:hash "7fd2d95e0f9239939598e7a9b8d5a273" :file
"/path/to/myfile.org.gpg" :inode 41551881)
     :expiry 30)

Please, please, be reasonable about this. Please, do not store
information about known encrypted files in other places. Please, allow
users to disable the feature cleanly and safely for arbitrary files if
they choose to.

Best,
Gustavo.

Re: How to disable org-persist in a given file?

[Ihor Radchenko]

Gustavo Barros <gusbrs.2016@gmail.com> writes:

> For the record, even ".org.gpg" files generate an entry in the cache
> index. (True, not the `:persist-file' itself though).
>
> My ~/.cache/org-persist/index contains:
>
>     (:container
>      ((elisp org-element--headline-cache)
>       (elisp org-element--cache))
>      :persist-file "c8/fd2b62-45cc-41c8-8571-d944c76b1f15" :associated
>      (:hash "7fd2d95e0f9239939598e7a9b8d5a273" :file
> "/path/to/myfile.org.gpg" :inode 41551881)
>      :expiry 30)

I think `recentf-save-file' for example is no different. And
org-id-locations-file. And custom-file, if you happen to save safe
buffer-local variables by answering "!" in Emacs prompt. And many many
other places.

I do not think that file name, even from encrypted volume, is something
we need to worry about.

I even suspect that, for example, browser cache often contains all kinds
of secrets, like files associated with web pages were you logged in. And
they can be read by anyone familiar with the layout! (like
https://www.nirsoft.net/utils/chrome_cache_view.html)

> Please, please, be reasonable about this. Please, do not store
> information about known encrypted files in other places. Please, allow
> users to disable the feature cleanly and safely for arbitrary files if
> they choose to.

That said, do not worry about this issue being forgotten. But it is not
easy to design cleanly. I am thinking about it.
Of course, if you have good ideas or patches, they are welcome.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: How to disable org-persist in a given file?

[Gustavo Barros]

On Tue, 2 May 2023 at 17:54, Ihor Radchenko <yantar92@posteo.net> wrote:

> I think `recentf-save-file' for example is no different. And
> org-id-locations-file. And custom-file, if you happen to save safe
> buffer-local variables by answering "!" in Emacs prompt. And many many
> other places.

Those two are easy to avoid, if one wishes. But granted, there are
plenty of places, some keeping file names, some worse. Still, this is
arguably something to avoid.

> I do not think that file name, even from encrypted volume, is something
> we need to worry about.

Granted too. What I'm worried about is being able to disable the
feature. Besides, why store it in the index if the persist file does
not exist?

> I even suspect that, for example, browser cache often contains all kinds
> of secrets, like files associated with web pages were you logged in. And
> they can be read by anyone familiar with the layout! (like
> https://www.nirsoft.net/utils/chrome_cache_view.html)

I really hope that's not Org's benchmark.  ;-)

> That said, do not worry about this issue being forgotten. But it is not
> easy to design cleanly. I am thinking about it.
> Of course, if you have good ideas or patches, they are welcome.

Thank you!

Best,
Gustavo.

Re: How to disable org-persist in a given file?

[Ihor Radchenko]

Gustavo Barros <gusbrs.2016@gmail.com> writes:

> For the record, even ".org.gpg" files generate an entry in the cache
> index. (True, not the `:persist-file' itself though).
>
> My ~/.cache/org-persist/index contains:
>
>     (:container
>      ((elisp org-element--headline-cache)
>       (elisp org-element--cache))
>      :persist-file "c8/fd2b62-45cc-41c8-8571-d944c76b1f15" :associated
>      (:hash "7fd2d95e0f9239939598e7a9b8d5a273" :file
> "/path/to/myfile.org.gpg" :inode 41551881)
>      :expiry 30)

This was actually just an omission in `org-persist-gc'.
Now fixed on main.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=e11073d17

> Please, please, be reasonable about this. Please, do not store
> information about known encrypted files in other places. Please, allow
> users to disable the feature cleanly and safely for arbitrary files if
> they choose to.

But do not create so much sense of urgency here. As I explained, this is
not something that much critical to fix ASAP.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: How to disable org-persist in a given file?

[Gustavo Barros]

On Wed, 3 May 2023 at 07:55, Ihor Radchenko <yantar92@posteo.net> wrote:

> This was actually just an omission in `org-persist-gc'.
> Now fixed on main.
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=e11073d17

Thank you!

> But do not create so much sense of urgency here. As I explained, this is
> not something that much critical to fix ASAP.

Fair, I won't anymore.

Best,
Gustavo.